CN111754816B - Device, method, system, terminal and medium for identifying intention of mobile object - Google Patents
Device, method, system, terminal and medium for identifying intention of mobile object Download PDFInfo
- Publication number
- CN111754816B CN111754816B CN202010499374.8A CN202010499374A CN111754816B CN 111754816 B CN111754816 B CN 111754816B CN 202010499374 A CN202010499374 A CN 202010499374A CN 111754816 B CN111754816 B CN 111754816B
- Authority
- CN
- China
- Prior art keywords
- intention
- vehicle
- moving object
- target vehicle
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/167—Driving aids for lane monitoring, lane changing, e.g. blind spot detection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method, a system, a terminal and a storage medium for identifying the intention of a mobile object, which are used for detecting the mobile object in a traffic jam scene, identifying the driving intention of the approaching mobile object at the next moment, judging the probability of collision caused by the fact that the near-field mobile object cuts into a main planning path, and further giving a corresponding jam intention prompt to a driver at a proper time point, or selecting proper avoidance response according to the result of jam intention judgment, or iterating the self driving path again. The invention not only can judge the jam intention, but also can pre-judge the lane changing intention including the jam intention in front of the vehicle, the lane changing intention in front of the vehicle, the overtaking intention on the left side and the right side of the vehicle and the lane changing intention behind the vehicle, thereby providing preconditions for the prediction of the unmanned near-field vehicle behaviors of the L4 grade and the L5 grade.
Description
Technical Field
The present invention relates to the field of automotive electronics, and in particular, to a device, a method, a system, a terminal, and a medium for identifying intent of a moving object.
Background
Safety is a major factor in pulling the demand for unmanned vehicles to grow. In particular, in the case of traffic jams and complex-scene China traffic environments, traffic accidents caused by misjudgment of intention of drivers to near-field cut-in objects are counted every year.
The existing advanced auxiliary driving system also makes corresponding judgment based on the real-time perception of the sensor, and the judgment has certain postponement. It is due to this postponement that existing advanced driver assistance systems are unable to give the driver a corresponding intent hint at the appropriate point in time, or select the appropriate avoidance response based on the result of the intent determination, or iterate the own travel path again.
Disclosure of Invention
In order to solve the above and other potential technical problems, the present invention provides a device, a method, a system, a terminal and a medium for identifying a mobile object, so as to detect a mobile object in a traffic jam scene, identify a driving intention of an approaching mobile object at a next moment, determine a probability of collision caused by a near-field mobile object cutting into a main planning path, and further give a corresponding jam intention prompt to a driver at a proper time point, or select a proper avoidance response according to a result of jam intention determination, or iterate a self-driving path again. The invention not only can judge the jam intention, but also can pre-judge various intentions including the jam intention in front of the vehicle, the lane change intention in front of the vehicle, the overtaking intention on the left side and the right side of the vehicle, the lane change intention behind the vehicle and the like, and provides preconditions for the unmanned near-field vehicle behavior prediction of the L4 grade and the L5 grade.
A moving object intention recognition apparatus comprising:
an intention prediction module and an intention judgment module;
the intention prediction module comprises a recurrent neural network consisting of an input layer, an output layer and at least one neuron cell layer, wherein the input element of the input layer is each dimensional information representing the running state of the near-field moving object; the output element of the output layer is the probability of each near-field moving object intention prediction result;
the intention judging module is used for identifying the specific intention of the near-field moving object through algorithm processing by combining the result output by the intention predicting module with the driving state of the vehicle, the driving rule element of the scene where the vehicle is located and the security level setting element.
Further, the input elements of the input layer, which are used for representing the running state of the near-field mobile object, are time sequence data, namely, the input layer data collected at different time points, and the time sequence data of the input layer reflect the change state and/or degree of the running state of the near-field mobile object along with time.
Further, the input elements of the input layer include, but are not limited to, camera sensing data of all directions of the vehicle, millimeter wave radar sensing data of all directions of the vehicle, ultrasonic sensing data of all directions of the vehicle, laser radar sensing data of the vehicle, and data clusters of near-field moving objects of the vehicle represented by fusion correction results of one or more types of data of the vehicle.
Further, the output layer output element is a probability of whether each near-field moving object will be intentionally jammed at the next time, and the output layer numbers each near-field moving object and outputs a probability value of each near-field moving object to be intentionally jammed at the next time in a matrix form.
Further, the recurrent neural network is a deep recurrent neural network.
Further, the deep recurrent neural network comprises n neuron cell layers, namely, a first neuron cell layer and a second neuron cell layer … are respectively marked from an input layer to an output layer, wherein the input of the first neuron cell layer comprises a data cluster of a vehicle near-field moving object at the moment and cell memory data at the moment on the first neuron cell layer, the input of the second neuron cell layer is an output result of the first neuron cell layer and cell memory data at the moment on the second neuron cell layer, the input of the nth neuron cell layer is a probability that the output result of the nth neuron cell layer and the cell memory data at the moment on the nth neuron cell layer are respectively marked, the output of the nth neuron cell layer is an intention prediction result of each near-field moving object, each branch model is trained in parallel among the first neuron cell layer, the second neuron cell layer and the … nth neuron cell layer, and the parallel training results are aggregated, synchronized and/or updated in a synchronous and/or asynchronous mode parameter and are applied to each branch model.
Further, the working principle of the neuron cell layer is as follows: the neuron cell layer is like a conventional memory cell, and comprises an input layer, a memory cell with self-circulation connection, a forgetting gate and an output layer; the input layer may allow the incoming signal to change the state of memory of the cell or prevent it. On the other hand, the input layer may allow the state of the cell memory to affect or prevent other neurons. Including but not limited to two vectors: h (t) and c (t) ("c" stands for "cell"), h (t) is considered a short term state, which represents input from the next layer of neuronal cells, and c (t) is considered a long term state, which represents memory of the neuronal cells at the previous time, which can last from one time step to another. Recurrent neural networks can learn the long term state of memory content, i.e., cell memory can selectively regulate interactions between the cell memory (i.e., memory cells) itself and the external environment through the amnestic gates and/or memory gates of the neuronal cell layers. As a long-term state c (t-1) traverses the network from left to right, it first passes through a forgetting gate, loses some of the memory cell memory at the last moment, and then adds some new cell memory addition (adding memory selected by the input gate) at the current moment. Therefore, in the continuous time axis, every time the input layer is input, some memories are discarded and some memories are added. Also, after addition, the long-term state is replicated and passed through the tanh function (i.e., g (t)), and the result is filtered by the output layer. This results in a short-term state h (t).
Further, the function of the fully-connected layer of the neuron cell layer is as follows: the input vector x (t) of the current input layer and the previous short-term state h (t-1) are fed to four different fully connected layers. Four fully attached layers all have different uses: the second fully connected layer is the layer outputting g (t). It has the effect of analysing the current input x (t) and the previous (short term) state h (t-1). In the cell layer of a conventional recurrent neural network, its outputs are directly output to y (t) and h (t). In long term memory neural networks (LSTM), the output of this h (t) is not directly output, but the direct output is stored in a long term state. The first full-connection layer, the third full-connection layer and the fourth full-connection layer are all gate controllers. Because they use logistical activation functions, their output ranges from 0 to 1. Their outputs are fed to the multiplication section so if they output 0 they will close the gate and if they output 1 they open the gate. The first fully connected layer controlled forget gate (controlled by f (t)) controls which part of the long term state should be forgotten. The input gate of the third full link layer control (controlled by i (t)) controls which part of g (t) of the second full link layer control should be added to the long term state. Finally, the output gate of the fourth fully connected layer (controlled by o (t)) controls which parts of the long term should read and output states at this time step (from h (t)) and y (t). In summary, the long-term memory neural network unit can learn to recognize important inputs by means of the action of the input gate and store them in a long-term state, forget unnecessary parts according to the action of the forgetting gate, memorize necessary parts, and learn to extract it as needed. They can be used to capture a time series, long text, audio recordings, and interesting portions of the input vector x (t) of the input layer in successive video frames.
Further, the specific intention judged by the intention judgment module includes, but is not limited to, a vehicle front jam intention, a lane change intention in front of the vehicle, a vehicle left side and right side overtaking intention, and a lane change intention behind the vehicle.
Further, when the intention judging module is combined with the vehicle running state, the vehicle running state comprises the relative position between the target vehicle and the vehicle at the current moment, the speed of the target vehicle, the acceleration of the target vehicle and the course angle of the target vehicle; the relative position between the target vehicle and the host vehicle, the speed of the target vehicle, the acceleration of the target vehicle, and the heading angle of the target vehicle in a continuous period of time before the current time of the target vehicle.
Further, when the intention judging module combines the driving rule elements of the scene where the vehicle is located, the driving rule elements of the scene where the vehicle is located include speed limit rules in the current scene, road traffic rule data packets in the current scene map, simulated traffic rule data packets virtually set in the current scene map, historical traffic records of the current scene map, and records recorded by weather and environment changes of the historical traffic records of the current scene map.
Further, when the intention judging module combines the security level setting element, the security level setting element is a record of traffic accidents caused by vehicle route planning made by adopting intention recognition in a history record of a current scene map, if the record exists, the credibility of vehicle intention recognition in the scene is reduced, and if the record does not exist, the credibility of vehicle intention recognition in the scene is improved.
Further, when the intention judging module identifies the specific intention of the near-field moving object through algorithm processing, the intention predicting module corrects the predicted probability by the intention predicting module according to the driving rule elements and the safety level setting elements of the scene where the vehicle is located, and then the corrected probability is selected to be displayed in a part exceeding the set rated alarm probability.
A moving object intention recognition method comprising the steps of:
s01: acquiring perception data, identifying near-field mobile objects in the perception data, and giving labels corresponding to each near-field mobile object; extracting the running state of the near-field moving object in different dimensions;
s02: taking each dimension representing the running state of the near-field mobile object as input, inputting an intention recognition network model, and outputting the probability of the intention prediction result of each near-field mobile object by the intention recognition network model;
s03: the probability of each near-field mobile object intention prediction result is combined with a running state of a vehicle, a running rule element of a scene where the vehicle is located and a security level setting element, and a near-field mobile object tag for identifying specific intention in the near-field mobile object is processed through an algorithm.
Further, step S04: the near field moving object of a specific intention is delivered to the driver in a manner including, but not limited to, image annotation, voice prompt.
Further, the step S01 further includes a step S011: data clusters related to the near-field mobile object are formed under labels of the near-field mobile object by classifying the running states of the near-field mobile object with multiple dimensions.
Further, the step S02 further includes a step S021: classifying the data in the data cluster according to the characterization dimension, and identifying the network model by the input intention.
A moving object intention recognition system comprising:
the sensing equipment comprises one or more of video sensing equipment, laser radar sensing equipment, millimeter wave sensing equipment and ultrasonic sensing equipment;
the sensing data preprocessing module comprises an image processing module, a millimeter wave processing module, an ultrasonic processing module and a laser radar processing module; the image processing module comprises a unified neural network for extracting a pre-selected frame from the video perception data and a neural network for screening a region of interest; the millimeter wave processing module, the ultrasonic processing module and the laser radar processing module are used for extracting the motion trail characteristics of the near-field moving object;
the intention prediction module comprises a recurrent neural network consisting of an input layer, an output layer and at least one neuron cell layer, wherein the input elements of the input layer are output data of the perception data preprocessing module, namely, each dimension information representing the running state of the near-field mobile object; the output element of the output layer is the probability of each near-field moving object intention prediction result;
the intention judging module is used for identifying the specific intention of the near-field moving object through algorithm processing by combining the result output by the intention predicting module with the driving state of the vehicle, the driving rule element of the scene where the vehicle is located and the security level setting element.
Further, the perception data preprocessing module further comprises a data fusion module, wherein the data fusion module is used for correcting and fusing data used for representing the running state of each near-field moving object.
Further, an image processing module in the perception data preprocessing module performs feature extraction and preprocessing on original video data acquired by a camera through a unified convolutional neural network in combination with algorithms of deep learning and an optical flow field.
Further, the millimeter wave processing module, the ultrasonic processing module and the laser radar processing module in the perception data preprocessing module need to firstly decode and cluster the data acquired by millimeter waves, ultrasonic waves and laser radars through the processing provided by corresponding sensor manufacturers, output the result of object tracking and detection, decode through a time domain processing neural network and extract the motion trail characteristics of other obstacles.
Further, the feature parts extracted by the image processing module, the millimeter wave processing module, the ultrasonic processing module and the laser radar processing module after preprocessing are uniformly input into a time domain recurrent neural network (LSTM NN) for comprehensive decision processing.
A terminal device such as a smart phone that can execute the above-described moving object intention recognition method program or a vehicle-mounted terminal control device that can execute the above-described moving object intention recognition method program.
A server comprising means for implementing the above-described mobile object intention recognition method and/or mobile object intention recognition system.
A computer storage medium storing a software program and/or a moving object intention recognition system corresponding to the moving object intention recognition method.
As described above, the present invention has the following advantageous effects:
the method comprises the steps of detecting a moving object in a traffic jam scene, identifying the driving intention of an approaching moving object at the next moment, judging the probability of collision caused by the fact that the near-field moving object cuts into a main planning path, and further giving a corresponding jam intention prompt to a driver at a proper time point, or selecting a proper avoidance reaction according to a jam intention judging result, or iterating the self driving path again. The invention can judge the jam intention, and can also pre-judge various intentions such as the jam intention in front of the vehicle, the lane change intention in front of the vehicle, the overtaking intention on the left side and the right side of the vehicle, the lane change intention behind the vehicle and the like.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a neuronal cell layer according to the present invention.
Fig. 2 is a schematic diagram of the deep-loop neural network of the present invention.
Fig. 3 is a schematic diagram showing training of the deep cyclic neural network of the present invention.
Fig. 4 is a schematic diagram of the intention prediction module and the intention judgment module.
Fig. 5 shows a flow chart of the present invention.
Fig. 6 shows a flow chart of another embodiment of the present invention.
Fig. 7 shows a flowchart of the process of the intention recognition system for a moving object.
Fig. 8 shows a schematic view of a scene identified for pinch-in intention when a vehicle is cut in.
Fig. 9 shows a schematic view of a scene intended to be identified at the end of a vehicle cut-in pinch.
Detailed Description
Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict.
It should be understood that the structures, proportions, sizes, etc. shown in the drawings are for illustration purposes only and should not be construed as limiting the invention to the extent that it can be practiced, since modifications, changes in the proportions, or otherwise, used in the practice of the invention, are not intended to be critical to the essential characteristics of the invention, but are intended to fall within the spirit and scope of the invention. Also, the terms such as "upper," "lower," "left," "right," "middle," and "a" and the like recited in the present specification are merely for descriptive purposes and are not intended to limit the scope of the invention, but are intended to provide relative positional changes or modifications without materially altering the technical context in which the invention may be practiced.
With reference to figure 4 of the drawings,
a moving object intention recognition apparatus comprising:
an intention prediction module and an intention judgment module;
the intention prediction module comprises a recurrent neural network consisting of an input layer, an output layer and at least one neuron cell layer, wherein the input element of the input layer is each dimensional information representing the running state of the near-field moving object; the output element of the output layer is the probability of each near-field moving object intention prediction result;
the intention judging module is used for identifying the specific intention of the near-field moving object through algorithm processing by combining the result output by the intention predicting module with the driving state of the vehicle, the driving rule element of the scene where the vehicle is located and the security level setting element.
As a preferred embodiment, the input elements of the input layer, which each represent the running state of the near-field mobile object, are time series data, that is, input layer data collected at different time points, and the time series data of the input layer reflect the time-varying state and/or degree of the running state of the near-field mobile object.
As a preferred embodiment, the input elements of the input layer include, but are not limited to, camera sensing data of all directions of the vehicle, millimeter wave radar sensing data of all directions of the vehicle, ultrasonic sensing data of all directions of the vehicle, laser radar sensing data of the vehicle, and data clusters of near-field moving objects of the vehicle represented by fusion correction results of one or more types of data thereof.
As a preferred embodiment, the output layer output element is a probability of whether each near-field moving object will be intentionally jammed at the next time, and the output layer numbers each near-field moving object and outputs a probability value of each near-field moving object to be intentionally jammed at the next time in a matrix form.
As a preferred embodiment, the recurrent neural network is a deep recurrent neural network.
As a preferred embodiment, the deep recurrent neural network includes n neuron cell layers, namely, a first neuron cell layer and a second neuron cell layer …, from an input layer to an output layer, where the input of the first neuron cell layer includes a data cluster of a near-field moving object of the vehicle at the present time and cell memory data at a time on the first neuron cell layer, the input of the second neuron cell layer is an output result of the first neuron cell layer and cell memory data at a time on the second neuron cell layer, the input of the nth neuron cell layer is cell memory data at a time on an output result of the n-1 th neuron cell layer and an output of the nth neuron cell layer, and the output of the nth neuron cell layer is a probability of an intention prediction result of each near-field moving object. Referring to fig. 3, each of the branch models is trained in parallel between the first, second, and … nth neuronal cell layers, and the parallel training results are aggregated and synchronously and/or asynchronously updated with model parameters and applied in each of the branch models. The neural network of the first neuron cell layer, the neural network of the second neuron cell layer and the neural network of the third neuron cell layer correspond to A, B, C three parallel branch parts, the network model training part is as shown in fig. 3, each branch is trained in parallel among the neuron cell layers, and then the branches are subjected to aggregation gradual change, updating and synchronization/asynchronization to the network model.
Referring to fig. 1, as a preferred embodiment, the neuronal cell layer works as follows: the neuron cell layer is like a conventional memory cell, and comprises an input layer, a memory cell with self-circulation connection, a forgetting gate and an output layer; the input layer may allow the incoming signal to change the state of memory of the cell or prevent it. On the other hand, the input layer may allow the state of the cell memory to affect or prevent other neurons. Including but not limited to two vectors: h (t) and c (t) ("c" stands for "cell"), h (t) is considered a short term state, which represents input from the next layer of neuronal cells, and c (t) is considered a long term state, which represents memory of the neuronal cells at the previous time, which can last from one time step to another. Recurrent neural networks can learn the long term state of memory content, i.e., cell memory can selectively regulate interactions between the cell memory (i.e., memory cells) itself and the external environment through the amnestic gates and/or memory gates of the neuronal cell layers. As a long-term state c (t-1) traverses the network from left to right, it first passes through a forgetting gate, loses some of the memory cell memory at the last moment, and then adds some new cell memory addition (adding memory selected by the input gate) at the current moment. Therefore, in the continuous time axis, every time the input layer is input, some memories are discarded and some memories are added. Also, after addition, the long-term state is replicated and passed through the tanh function (i.e., g (t)), and the result is filtered by the output layer. This results in a short-term state h (t).
Referring to fig. 1 to 2, as a preferred embodiment, the fully-connected layer of the neuronal cell layer functions as: the input vector x (t) of the current input layer and the previous short-term state h (t-1) are fed to four different fully connected layers. Four fully attached layers all have different uses: the second fully connected layer is the layer outputting g (t). It has the effect of analysing the current input x (t) and the previous (short term) state h (t-1). In the cell layer of a conventional recurrent neural network, its outputs are directly output to y (t) and h (t). In long term memory neural networks (LSTM), the output of this h (t) is not directly output, but the direct output is stored in a long term state. The first full-connection layer, the third full-connection layer and the fourth full-connection layer are all gate controllers. Because they use logistical activation functions, their output ranges from 0 to 1. Their outputs are fed to the multiplication section so if they output 0 they will close the gate and if they output 1 they open the gate. The first fully connected layer controlled forget gate (controlled by f (t)) controls which part of the long term state should be forgotten. The input gate of the third full link layer control (controlled by i (t)) controls which part of g (t) of the second full link layer control should be added to the long term state. Finally, the output gate of the fourth fully connected layer (controlled by o (t)) controls which parts of the long term should read and output states at this time step (from h (t)) and y (t). In summary, the long-term memory neural network unit can learn to recognize important inputs by means of the action of the input gate and store them in a long-term state, forget unnecessary parts according to the action of the forgetting gate, memorize necessary parts, and learn to extract it as needed. They can be used to capture a time series, long text, audio recordings, and interesting portions of the input vector x (t) of the input layer in successive video frames.
As a preferred embodiment, the specific intention judged by the intention judgment module includes, but is not limited to, a vehicle front jam intention, a vehicle front lane change intention, a vehicle left side and right side overtaking intention, and a vehicle rear lane change intention.
As a preferred embodiment, when the intention judging module is combined with a vehicle running state, the vehicle running state comprises a relative position between a target vehicle and a host vehicle at the current moment, a speed of the target vehicle, an acceleration of the target vehicle and a course angle of the target vehicle; the relative position between the target vehicle and the host vehicle, the speed of the target vehicle, the acceleration of the target vehicle, and the heading angle of the target vehicle in a continuous period of time before the current time of the target vehicle.
As a preferred embodiment, when the intention judging module combines the driving rule elements of the scene where the vehicle is located, the driving rule elements of the scene where the vehicle is located include speed limit rules in the current scene, road traffic rule data packets in the current scene map, simulated traffic rule data packets virtually set in the current scene map, historical traffic records of the current scene map, records recorded by weather and environmental changes of the historical traffic records of the current scene map.
As a preferred embodiment, when the intention judging module combines the security level setting element, the security level setting element is a record of traffic accidents caused by vehicle route planning made by adopting intention recognition in a history record of a current scene map, if the record exists, the credibility of vehicle intention recognition in the scene is reduced, and if the record does not exist, the credibility of vehicle intention recognition in the scene is improved.
In a preferred embodiment, when the intention judging module identifies the specific intention of the near-field moving object through algorithm processing, the intention predicting module corrects the probability predicted by the intention predicting module according to the driving rule element and the safety level setting element of the scene where the vehicle is located, and then the corrected probability is selected to be displayed in a part exceeding the set rated alarm probability.
Referring to fig. 5 to 7, a moving object intention recognition method includes the steps of:
s01: acquiring perception data, identifying near-field mobile objects in the perception data, and giving labels corresponding to each near-field mobile object; extracting the running state of the near-field moving object in different dimensions;
s02: taking each dimension representing the running state of the near-field mobile object as input, inputting an intention recognition network model, and outputting the probability of the intention prediction result of each near-field mobile object by the intention recognition network model;
s03: the probability of each near-field mobile object intention prediction result is combined with a running state of a vehicle, a running rule element of a scene where the vehicle is located and a security level setting element, and a near-field mobile object tag for identifying specific intention in the near-field mobile object is processed through an algorithm.
As a preferred embodiment, step S04 is further included: the near field moving object of a specific intention is delivered to the driver in a manner including, but not limited to, image annotation, voice prompt.
As a preferred embodiment, the step S01 further includes a step S011: data clusters related to the near-field mobile object are formed under labels of the near-field mobile object by classifying the running states of the near-field mobile object with multiple dimensions.
As a preferred embodiment, the step S02 further includes a step S021: classifying the data in the data cluster according to the characterization dimension, and identifying the network model by the input intention.
A moving object intention recognition system comprising:
the sensing equipment comprises one or more of video sensing equipment, laser radar sensing equipment, millimeter wave sensing equipment and ultrasonic sensing equipment;
the sensing data preprocessing module comprises an image processing module, a millimeter wave processing module and an ultrasonic processing module; the image processing module comprises a unified neural network for extracting a pre-selected frame from the video perception data and a neural network for screening a region of interest; the millimeter wave processing module and the ultrasonic processing module are used for acquiring the speed and the acceleration of the near-field moving object;
the intention prediction module comprises a recurrent neural network consisting of an input layer, an output layer and at least one neuron cell layer, wherein the input elements of the input layer are output data of the perception data preprocessing module, namely, each dimension information representing the running state of the near-field mobile object; the output element of the output layer is the probability of each near-field moving object intention prediction result;
the intention judging module is used for identifying the specific intention of the near-field moving object through algorithm processing by combining the result output by the intention predicting module with the driving state of the vehicle, the driving rule element of the scene where the vehicle is located and the security level setting element.
As a preferred embodiment, the sensing data preprocessing module further comprises a data fusion module, and the data fusion module is used for correcting and fusing data used for representing the running state of each near-field mobile object.
As a preferred embodiment, the image processing module in the perception data preprocessing module performs feature extraction and preprocessing on the original video data acquired by the camera through a unified convolutional neural network in combination with an algorithm of deep learning and an optical flow field.
As a preferred embodiment, the millimeter wave processing module, the ultrasonic processing module and the laser radar processing module in the perception data preprocessing module need to firstly process the data acquired by millimeter waves, ultrasonic waves and laser radars through decoding, clustering and the like provided by corresponding sensor manufacturers, output results of object tracking and detection, and then decode through a time domain processing neural network to extract the motion trail characteristics of other obstacles.
As a preferred embodiment, the feature parts extracted by the image processing module, the millimeter wave processing module, the ultrasonic processing module and the laser radar processing module after preprocessing are uniformly input into a time domain recurrent neural network (LSTM NN) for comprehensive decision processing.
Referring to fig. 8-9, the present invention detects a moving object in a traffic jam scene, identifies a driving intention of an approaching moving object at a next moment, determines a probability of collision caused by a near-field moving object cutting into a main planning path, and further gives a corresponding jam intention prompt to a driver at a proper time point, or selects a proper avoidance response according to a result of jam intention determination, or iterates a self-driving path again. The invention not only can judge the jam intention, but also can pre-judge various intentions including the jam intention in front of the vehicle, the lane change intention in front of the vehicle, the overtaking intention on the left side and the right side of the vehicle, the lane change intention behind the vehicle and the like, and provides preconditions for the unmanned near-field vehicle behavior prediction of the L4 grade and the L5 grade.
A terminal device such as a smart phone that can execute the above-described moving object intention recognition method program or a vehicle-mounted terminal control device that can execute the above-described moving object intention recognition method program.
A server comprising means for implementing the above-described mobile object intention recognition method and/or mobile object intention recognition system.
A computer storage medium storing a software program and/or a moving object intention recognition system corresponding to the moving object intention recognition method.
As a preferred embodiment, the present embodiment further provides a terminal device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack-mounted server, a blade server, a tower server, or a rack-mounted server (including an independent server, or a server cluster formed by a plurality of servers) that can execute a program, or the like. The terminal device of this embodiment includes at least, but is not limited to: a memory, a processor, and the like, which may be communicatively coupled to each other via a system bus. It should be noted that a terminal device having a component memory, a processor, but it should be understood that not all illustrated components are required to be implemented, and that alternative methods of moving object intent recognition may implement more or fewer components.
As a preferred embodiment, the memory (i.e., readable storage medium) includes flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of a computer device, such as a plug-in hard disk provided on the computer device, a Smart Media Card (SMC), a secure digital (SecureDigital, SD) Card, a Flash memory Card (Flash Card), etc. Of course, the memory may also include both internal storage units of the computer device and external storage devices. In this embodiment, the memory is typically used to store an operating system installed in the computer device and various types of application software, such as moving object intention recognition method program codes and the like in the embodiment. In addition, the memory can be used to temporarily store various types of data that have been output or are to be output.
A computer-readable storage medium having stored thereon a computer program, characterized by: the program when executed by the processor implements the steps in the moving object intention recognition method described above.
The present embodiment also provides a computer-readable storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor, performs the corresponding functions. The computer-readable storage medium of the present embodiment is for storing a moving object intention recognition method program, which when executed by a processor, implements the moving object intention recognition method in the moving object intention recognition method embodiment.
The above embodiments are merely illustrative of the principles of the present invention and its effectiveness, and are not intended to limit the invention. Modifications and variations may be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the invention. Accordingly, it is intended that all equivalent modifications and variations of the invention be covered by the claims of this invention, which are within the skill of those skilled in the art, be included within the spirit and scope of this invention.
Claims (12)
1. A moving object intention recognition apparatus, characterized by comprising:
an intention prediction module and an intention judgment module;
the intention prediction module comprises a recurrent neural network consisting of an input layer, an output layer and at least one neuron cell layer, wherein the input element of the input layer is each dimensional information representing the running state of the near-field moving object; the output element of the output layer is the probability of each near-field moving object intention prediction result;
the intention judging module combines the result output by the intention predicting module with a running rule element and a security level setting element comprising the running state of the vehicle, the scene where the vehicle is located, and recognizes the specific intention of the near-field moving object through algorithm processing; the vehicle running state comprises the relative position between the target vehicle and the vehicle at the current moment, the speed of the target vehicle, the acceleration of the target vehicle and the course angle of the target vehicle; the relative position between the target vehicle and the host vehicle, the speed of the target vehicle, the acceleration of the target vehicle and the course angle of the target vehicle in a continuous time period before the current moment of the target vehicle;
the specific intention judged by the intention judgment module comprises a vehicle front jam intention, a vehicle front lane change intention, a vehicle left side and right side overtaking intention and a vehicle rear lane change intention.
2. The mobile object intention recognition apparatus according to claim 1, wherein the input elements of the input layer input each representing a running state of the near-field mobile object are time series data, i.e., input layer data collected at different points of time, the input layer time series data reflecting a time-varying state and/or degree of the running state of the near-field mobile object.
3. The moving object intention recognition apparatus according to claim 1, wherein the output layer outputs a probability that the respective near-field moving object will be intentionally jammed at the next time as an element, the output layer numbering the respective near-field moving object and outputting a probability value of the next intentional jam of each near-field moving object in a matrix form.
4. The moving object intention recognition device according to claim 1, wherein the recurrent neural network is a deep recurrent neural network, the deep recurrent neural network includes n-th neuron cell layers, namely, a first neuron cell layer and a second neuron cell layer … -th neuron cell layer, from an input layer to an output layer, the input of the first neuron cell layer includes a data cluster of a vehicle near-field moving object at the present moment and cell memory data at a moment on the first neuron cell layer, the input of the second neuron cell layer is an output result of the first neuron cell layer and cell memory data at a moment on the second neuron cell layer, the input of the n-th neuron cell layer is cell memory data at a moment on the n-1-th neuron cell layer and an output of the n-th neuron cell layer is probability of an intention prediction result of each near-field moving object.
5. A method for identifying the intention of a moving object, comprising the steps of:
s01: acquiring perception data, identifying near-field mobile objects in the perception data, and giving labels corresponding to each near-field mobile object; extracting the running state of the near-field moving object in different dimensions;
s02: taking each dimension representing the running state of the near-field mobile object as input, inputting an intention recognition network model, and outputting the probability of the intention prediction result of each near-field mobile object by the intention recognition network model;
s03: combining the probability of each near-field mobile object intention prediction result with a running rule element and a security level setting element of a scene where a vehicle is located, and processing a near-field mobile object tag for identifying a specific intention in the near-field mobile object through an algorithm; the vehicle running state comprises the relative position between the target vehicle and the vehicle at the current moment, the speed of the target vehicle, the acceleration of the target vehicle and the course angle of the target vehicle; the relative position between the target vehicle and the host vehicle, the speed of the target vehicle, the acceleration of the target vehicle and the course angle of the target vehicle in a continuous time period before the current moment of the target vehicle;
the specific intention judged by the intention judgment module comprises a vehicle front jam intention, a vehicle front lane change intention, a vehicle left side and right side overtaking intention and a vehicle rear lane change intention.
6. The moving object intention recognition method according to claim 5, characterized by further comprising step S04: the near field moving object with specific intention is transmitted to the driver in a mode of comprising image annotation and voice prompt.
7. The moving object intention recognition method according to claim 6, wherein the step S01 further comprises a step S011: classifying the running state of the near-field mobile object with multiple dimensions under the label of the near-field mobile object to form a data cluster related to the near-field mobile object; the step S02 further includes a step S021: classifying the data in the data cluster according to the characterization dimension, and inputting the data into the intention recognition network model.
8. A moving object intention recognition system, characterized by comprising:
the sensing equipment comprises one or more of video sensing equipment, laser radar sensing equipment, millimeter wave sensing equipment and ultrasonic sensing equipment;
the sensing data preprocessing module comprises an image processing module, a millimeter wave processing module and an ultrasonic processing module; the image processing module comprises a unified neural network for extracting a pre-selected frame from the video perception data and a neural network for screening a region of interest; the millimeter wave processing module and the ultrasonic processing module are used for acquiring the speed and the acceleration of the near-field moving object;
the intention prediction module comprises a recurrent neural network consisting of an input layer, an output layer and at least one neuron cell layer, wherein the input elements of the input layer are output data of the perception data preprocessing module, namely, each dimension information representing the running state of the near-field mobile object; the output element of the output layer is the probability of each near-field moving object intention prediction result;
the intention judging module combines the result output by the intention predicting module with a running rule element and a security level setting element comprising the running state of the vehicle, the scene where the vehicle is located, and identifies the specific intention of the near-field moving object through algorithm processing; the vehicle running state comprises the relative position between the target vehicle and the vehicle at the current moment, the speed of the target vehicle, the acceleration of the target vehicle and the course angle of the target vehicle; the relative position between the target vehicle and the host vehicle, the speed of the target vehicle, the acceleration of the target vehicle and the course angle of the target vehicle in a continuous time period before the current moment of the target vehicle;
the specific intention judged by the intention judgment module comprises a vehicle front jam intention, a vehicle front lane change intention, a vehicle left side and right side overtaking intention and a vehicle rear lane change intention.
9. The mobile object intention recognition system of claim 8, wherein the perception data preprocessing module further comprises a data fusion module for modifying and fusing data for characterizing the operational state of each near-field mobile object.
10. The mobile object intention recognition system according to claim 8, wherein the image processing module in the perception data preprocessing module performs feature extraction and preprocessing on the original video data acquired by the camera through a unified convolutional neural network by combining with an algorithm of deep learning and an optical flow field; the millimeter wave processing module, the ultrasonic processing module and the laser radar processing module in the perception data preprocessing module need to decode and cluster data acquired by millimeter waves, ultrasonic waves and laser radars through corresponding sensor manufacturers, output results of object tracking and detection, decode through a time domain processing neural network and extract movement track characteristics of other obstacles; the characteristic parts extracted by the image processing module, the millimeter wave processing module, the ultrasonic processing module and the laser radar processing module after pretreatment are uniformly input into a time domain recurrent neural network (LSTM NN) for comprehensive decision processing.
11. A terminal device, characterized by: the terminal device is a smart phone controlling the mobile object intention recognition method according to any one of the above claims 5 to 7 or a vehicle-mounted terminal control device executing the mobile object intention recognition method according to any one of the above claims 5 to 7.
12. A computer-readable storage medium having stored thereon a computer program, characterized by: the program, when executed by a processor, implements the steps of the method of any of claims 5 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010499374.8A CN111754816B (en) | 2020-06-04 | 2020-06-04 | Device, method, system, terminal and medium for identifying intention of mobile object |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010499374.8A CN111754816B (en) | 2020-06-04 | 2020-06-04 | Device, method, system, terminal and medium for identifying intention of mobile object |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111754816A CN111754816A (en) | 2020-10-09 |
| CN111754816B true CN111754816B (en) | 2023-04-28 |
Family
ID=72674500
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010499374.8A Active CN111754816B (en) | 2020-06-04 | 2020-06-04 | Device, method, system, terminal and medium for identifying intention of mobile object |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111754816B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153207B (en) * | 2021-11-29 | 2024-02-27 | 北京三快在线科技有限公司 | Control method and control device of unmanned equipment |
| CN114973166B (en) * | 2022-07-26 | 2022-10-25 | 中诚华隆计算机技术有限公司 | Traffic information tracking method, system and computer equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1170652A1 (en) * | 2000-07-06 | 2002-01-09 | Renault | Process of adaptive regulation of the distance between two mobile vehicles |
| CN110758382A (en) * | 2019-10-21 | 2020-02-07 | 南京航空航天大学 | Surrounding vehicle motion state prediction system and method based on driving intention |
Family Cites Families (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4281543B2 (en) * | 2003-12-16 | 2009-06-17 | 日産自動車株式会社 | VEHICLE DRIVE OPERATION ASSISTANCE DEVICE AND VEHICLE HAVING VEHICLE DRIVE OPERATION ASSISTANCE DEVICE |
| CN100492437C (en) * | 2007-06-01 | 2009-05-27 | 清华大学 | Quick identification method for object vehicle lane changing |
| US10611371B2 (en) * | 2017-09-14 | 2020-04-07 | Toyota Motor Engineering & Manufacturing North America, Inc. | System and method for vehicle lane change prediction using structural recurrent neural networks |
| CN107919027B (en) * | 2017-10-24 | 2020-04-28 | 北京汽车集团有限公司 | Method, device and system for predicting lane change of vehicle |
| KR102463720B1 (en) * | 2017-12-18 | 2022-11-07 | 현대자동차주식회사 | System and Method for creating driving route of vehicle |
| US20190367019A1 (en) * | 2018-05-31 | 2019-12-05 | TuSimple | System and method for proximate vehicle intention prediction for autonomous vehicles |
| CN110210058B (en) * | 2019-04-26 | 2024-04-26 | 纵目科技（上海）股份有限公司 | Reference line generation method, system, terminal and medium conforming to vehicle dynamics |
| CN110097785B (en) * | 2019-05-30 | 2022-06-07 | 长安大学 | Recognition early warning device and early warning method for front vehicle cut-in or emergency lane change |
| CN110555402A (en) * | 2019-08-27 | 2019-12-10 | 北京纵目安驰智能科技有限公司 | congestion car following method, system, terminal and storage medium based on look-around |
| CN110796856B (en) * | 2019-10-16 | 2022-03-25 | 腾讯科技（深圳）有限公司 | Vehicle lane change intention prediction method and training method of lane change intention prediction network |
| CN111079590B (en) * | 2019-12-04 | 2023-05-26 | 东北大学 | Peripheral vehicle behavior pre-judging method of unmanned vehicle |
| CN111104969B (en) * | 2019-12-04 | 2023-09-19 | 东北大学 | Collision possibility pre-judging method for unmanned vehicle and surrounding vehicles |
| CN111114556A (en) * | 2019-12-24 | 2020-05-08 | 北京工业大学 | Lane change intention identification method based on LSTM under multi-source exponential weighting loss |
-
2020
- 2020-06-04 CN CN202010499374.8A patent/CN111754816B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1170652A1 (en) * | 2000-07-06 | 2002-01-09 | Renault | Process of adaptive regulation of the distance between two mobile vehicles |
| CN110758382A (en) * | 2019-10-21 | 2020-02-07 | 南京航空航天大学 | Surrounding vehicle motion state prediction system and method based on driving intention |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111754816A (en) | 2020-10-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107368890B (en) | Road condition analysis method and system based on deep learning and taking vision as center | |
| CN108388834A (en) | The object detection mapped using Recognition with Recurrent Neural Network and cascade nature | |
| CN109711557B (en) | Driving track prediction method, computer equipment and storage medium | |
| US9527384B2 (en) | Driving context generation system for generating driving behavior description information | |
| US8965115B1 (en) | Adaptive multi-modal detection and fusion in videos via classification-based-learning | |
| US20220048536A1 (en) | Method and device for testing a driver assistance system | |
| Arnelid et al. | Recurrent conditional generative adversarial networks for autonomous driving sensor modelling | |
| US11436744B2 (en) | Method for estimating lane information, and electronic device | |
| Dong et al. | A hybrid spatial–temporal deep learning architecture for lane detection | |
| CN111754816B (en) | Device, method, system, terminal and medium for identifying intention of mobile object | |
| US10878276B2 (en) | Method and system for detecting change of context in video streams | |
| CN112085165A (en) | Decision information generation method, device, equipment and storage medium | |
| US11086299B2 (en) | System and method for estimating uncertainty of the decisions made by a supervised machine learner | |
| US20200089966A1 (en) | Recognizing fine-grained objects in surveillance camera images | |
| CN112307978A (en) | Target detection method and device, electronic equipment and readable storage medium | |
| US11760387B2 (en) | Driving policies determination | |
| CN114118349A (en) | Method, system and apparatus for user understandable interpretable learning models | |
| Maye et al. | Bayesian on-line learning of driving behaviors | |
| Sayeed et al. | Bangladeshi traffic sign recognition and classification using cnn with different kinds of transfer learning through a new (btsrb) dataset | |
| CN111753371B (en) | Training method, system, terminal and storage medium for vehicle body control network model | |
| CN116461507A (en) | Vehicle driving decision method, device, equipment and storage medium | |
| Zhang et al. | A general framework of learning multi-vehicle interaction patterns from video | |
| Meftah et al. | Deep residual network for autonomous vehicles obstacle avoidance | |
| EP4050510A1 (en) | Object information calculation method and system | |
| CN114889608A (en) | Attention mechanism-based vehicle lane change prediction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |