CN114926825A - Vehicle driving behavior detection method based on space-time feature fusion - Google Patents

Vehicle driving behavior detection method based on space-time feature fusion Download PDF

Info

Publication number
CN114926825A
CN114926825A CN202210547632.4A CN202210547632A CN114926825A CN 114926825 A CN114926825 A CN 114926825A CN 202210547632 A CN202210547632 A CN 202210547632A CN 114926825 A CN114926825 A CN 114926825A
Authority
CN
China
Prior art keywords
driving
driving behavior
data
time
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210547632.4A
Other languages
Chinese (zh)
Inventor
宋梁
刘靖
朱伟
张冠华
蒋林华
胡兴
商慧亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fudan University
Original Assignee
Fudan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fudan University filed Critical Fudan University
Priority to CN202210547632.4A priority Critical patent/CN114926825A/en
Publication of CN114926825A publication Critical patent/CN114926825A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/30—Noise filtering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/52—Scale-space analysis, e.g. wavelet analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/62—Extraction of image or video features relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/80—Technologies aiming to reduce greenhouse gasses emissions common to all road transportation technologies
    • Y02T10/84—Data processing systems or methods, management, administration

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Traffic Control Systems (AREA)

Abstract

The invention belongs to the technical field of vehicle driving detection, and particularly relates to a vehicle driving behavior detection method based on space-time feature fusion. The method comprises the following steps: in an off-line stage, data preprocessing is carried out, space and time characteristics are respectively extracted through a space-time characteristic extraction network, and driving context information is embedded into a full connection layer; using a Softmax function to classify the driving behaviors of the fused space-time characteristics; in the online stage, driving behavior grading is carried out on the detected driving behaviors, and a grading strategy firstly uses a driving behavior detection model to automatically detect specific driving behaviors; then, the daily and long-term driving behaviors of the driver are comprehensively evaluated by combining two modes of driving performance scores and driving level integrals, so that the driver can be better guided to evolve towards efficient driving and safe driving; and finally, the driving data and a traffic management department can be selected to be networked, and the traffic management and the safety are optimized. And driving behavior detection and feedback are carried out through the smart phone, so that the application value of the smart phone is greatly improved.

Description

Vehicle driving behavior detection method based on space-time feature fusion
Technical Field
The invention belongs to the technical field of vehicle driving detection, and particularly relates to a vehicle driving behavior detection method.
Background
The driving behavior detection aims at detecting dangerous driving events through a detection algorithm, and is widely applied to the fields of traffic management, automobile insurance, fuel consumption optimization and the like. The driving behavior detection method may be classified into a conventional machine learning method and a deep learning method. The former is limited by human domain knowledge, meaningful features need to be selected manually in the feature extraction stage, and the latter can automatically extract time and space relations in data by designing a neural network, so that the method is widely applied to the fields of water quality prediction, quality detection and the like. However, deep learning models such as CNN alone can only extract spatial features, LSTM is not effective in capturing long-range dependencies, and they all treat all features equally.
Disclosure of Invention
The invention aims to provide a vehicle driving behavior detection method based on space-time feature fusion, which enables a neural network to learn space information and time information simultaneously so as to improve the detection capability of dangerous driving events. The invention can detect 7 driving behaviors: sudden braking, quick lane change, continuous lane change, quick left turn, quick right turn, quick turn around and normal driving.
The vehicle driving behavior detection method based on the spatio-temporal feature fusion mainly comprises an off-line training stage and an on-line training stage. The off-line training stage comprises data preprocessing, space-time feature extraction and driving classification. The trained model can efficiently and automatically detect driving behavior in multi-modal sensory information within a sliding window. The on-line detection stage comprises real-time data processing, driving behavior detection and driving behavior evaluation. The offline trained model is loaded through a software program of the smart phone, so that real-time driving behavior detection and driving behavior evaluation are realized, and the method is shown in fig. 2. The method comprises the following specific steps:
(1) data preprocessing (L1)
Collecting vehicle multimodal sensing offline data by a smartphone, the data comprising: vehicle acceleration and sensing data such as a gyroscope, GPS data, a magnetic sensor, road types, weather types and the like, and preprocessing the data, including coordinate conversion, up-sampling (aiming at low sampling rate modal data), wavelet transformation denoising, maximum and minimum normalization, sliding window division and the like;
(2) spatiotemporal feature extraction (L2)
Respectively extracting the characteristics of the spatial and temporal relations of different modal data through a CNN (deep convolutional neural network) and an LSTM (long short term memory network), and respectively carrying out weight measurement on the spatial characteristics and the temporal characteristics through a self-attention network;
(3) classification of Driving behavior (L3)
Classifying the space-time characteristics by using a fully-connected network layer and a Softmax function to obtain the classification of specific driving behaviors;
(4) real-time data processing (S1)
Obtain the power data of vehicle in real time by smart mobile phone APP application, include: vehicle acceleration, direction, magnetic sensor, GPS data, driving context information (such as weather type, road type, map information and the like), and pre-processing real-time multi-modal data;
(5) detecting driving behavior (S2)
Detecting specific driving behavior classes by using a driving behavior classification model trained offline;
(6) evaluation of Driving behavior (S3)
Evaluating the driving behavior by adopting two evaluation rules; firstly, evaluating the driving performance score for standardizing the daily driving performance of a driver; second, the driving level integral evaluation is used for improving the driving habit of the driver;
the deduction formula of the driving performance score is as follows:
score＝100-count ano (1)
wherein, count ano Counting 1 time when abnormal driving behaviors occur for 5 times or more in every 10 kilometers, and performing voice warning; the abnormal driving rows are divided into six types, which are respectively: quick braking, quick lane change, quick continuous lane change, quick left turn, quick right turn and quick turn around.
In the invention, the specific flow of the data preprocessing in the step (1) is as follows:
first, consider that the raw signal is obtained by multiple smartphone sensors, where different sensors have different frequencies; therefore, the low-frequency signals need to be up-sampled, and the up-sampling operation of the original signals is expressed by linear interpolation filtering, which is expressed as follows:
Figure BDA0003639074800000021
wherein os (-) and s (-) represent the up-sampled and original signals, respectively; i all right angle 2 And i 1 The rear and front positions of the position i are up-sampled, respectively.
Then, the sampled signals are decomposed through wavelet transformation to obtain smoother sequences, so that the generalization capability of the model is improved. The wavelet transform operation is represented as:
Figure BDA0003639074800000022
wherein, N, l m And N w Respectively representing the length of the original sequence, the number of decomposed layers and the length of a decomposition filter, and finally obtaining a sequence after denoising through inverse wavelet transform.
Due to the different modal input signals having different value ranges. Once the raw data is fed into the model, the training speed is slowed and detection performance may be affected. The present invention utilizes max-min normalization to map all variables to the range [0,1] to mitigate the effects. Expressed as:
Figure BDA0003639074800000023
wherein, s', s min And s max Respectively representing the minimum value and the maximum value of the original variable, the normalized variable and the original variable to be normalized.
Finally, in order to explore the original features of the driving behavior to the maximum extent, the invention divides the preprocessed sequence into a plurality of input sequences, and inputs the input sequences into the network through a sliding window strategy, which can be expressed as:
Figure BDA00036390748000000316
wherein ts, ws and T are the duration of a single time step, the duration of a sliding window and the number of time steps, s, respectively w And f hz Respectively the total sequence length of a single sliding window and the sampling rate of the data set.
In the invention, the specific process of the spatio-temporal feature extraction in the step (2) is as follows:
first, spatial features in driving behavior data are extracted through a CNN network, and are expressed as:
Figure BDA0003639074800000031
Figure BDA0003639074800000032
Figure BDA0003639074800000033
wherein v is tm Is c tm By means of the implicit representation of the full connection layer calculation,
Figure BDA0003639074800000034
and
Figure BDA0003639074800000035
are network parameters that are automatically learned in the tth time step in the CNN network,
Figure BDA0003639074800000036
is a normalized weight calculated by the Softmax function,
Figure BDA0003639074800000037
is a unified representation of all modalities after calculation with attention.
Then, the time characteristics in the driving behavior data are extracted through the LSTM network, and are expressed as follows:
Figure BDA0003639074800000038
Figure BDA0003639074800000039
Figure BDA00036390748000000310
Figure BDA00036390748000000311
Figure BDA00036390748000000312
wherein i t 、f t 、o t 、
Figure BDA00036390748000000313
And C t Respectively an input gate, a forgetting gate, an output gate, a candidate memory cell and a memory cell of the LSTM.
Figure BDA00036390748000000314
And h t-1 The hidden representation output at the tth time step of the CNN network (also the input at the tth time step of the LSTM network) and the hidden state at the t-1 time of the LSTM network are respectively. W and b represent the network parameters that the above-described gating units in the LSTM network automatically learn, respectively. The extracted temporal features are further weighted by the fully connected layer and the Softmax function for the importance of different time steps, expressed as:
k t ＝tanh(W 2 h t +b 2 )， (14)
Figure BDA00036390748000000315
δ＝∑ t β t h t ， (16)
wherein k is t 、β t And delta respectively represents the hidden representation of LSTM hidden state obtained after the calculation of the full connection layer, the normalized weight of the time characteristic and the unified representation of all time steps after the attention calculation. h is t Is the input at t time step of the full connection layer (also the output of LSTM at t time step), w 2 、W 2 And b 2 Respectively, representing automatically learned network parameters in a fully connected network.
In the invention, the specific process of classifying the driving behaviors in the step (3) comprises the following steps:
embedding driving context information into a full connection layer, classifying driving behaviors, and expressing as:
e r ＝onehot(v)×W e ， (17)
Figure BDA0003639074800000041
Figure BDA0003639074800000042
wherein e is r 、
Figure BDA0003639074800000043
And r behavior Respectively representing embedded driving context information, a driving behavior classification calculated by a Softmax function, and a specific driving behavior class. onehot (v) and f s The unique variables of the driving context information v and the hidden expressions obtained by the previous layer after the output delta of the attention network is calculated again by using the full connection (the process is the same as the formula (14)) are respectively represented. argmax (·) is the driving behavior that solves for the highest probability in the predicted driving behavior category. W is a group of e 、W y And b y Network parameters that represent classification layer network auto-learning.
In the present invention, the driving behavior is evaluated in step (6), wherein:
the driving performance is divided into 5 grades, the initial value is 100 grades, and the driving performance grades are respectively divided into a table 1. When the driving behavior is normal driving and the score is below 100 points, 1 point is added every 10 kilometers. When the driving performance is graded to be 2, namely the driving consciousness is extremely poor, the driver is not recommended to drive on the highway section; when the driving performance is graded to 1, that is, the driving consciousness is poor, the driver is not recommended to travel on the urban road section.
The driving level is divided into 7 grades according to the integral, the specific grades are divided in the table 2, the initial score is 750, and the grade is 1. When the driving performance grade is 4 or more, adding 1 integral to the integral grade (the 1 st column in the table 2) of the current driving level corresponding to the integral grade (the 3 rd column in the table 2) of the current driving level; and 5 points are deducted only when 5 times or more of abnormal driving behaviors are accumulated in every 100 kilometers of driving.
TABLE 1 Driving Performance Classification
Performance grade Score segment Description of the preferred embodiment
5 100 Excellent driving awareness
4 [90,99] Good driving consciousness
3 [80,89] Poor driving consciousness
2 [60,79] Poor consciousness of driving
1 <＝59 Bad driving consciousness
TABLE 2 Driving horizon Scoring
Figure BDA0003639074800000044
Figure BDA0003639074800000051
The invention provides a vehicle driving behavior detection method based on spatio-temporal feature fusion, which enables data to better meet the input requirement of a neural network through data preprocessing in an off-line stage; extracting spatial and temporal characteristics through a spatio-temporal characteristic extraction network, respectively measuring the importance of the spatial and temporal characteristics at different stages, and embedding driving context information in a subsequent full connection layer; the fused spatiotemporal features are classified for driving behavior by using a Softmax function. In the online stage, driving behavior scoring is carried out on the detected driving behavior, and compared with a scoring strategy set based on an acceleration or azimuth data threshold value in the prior art, the evaluation strategy of the invention firstly uses a driving behavior detection model to automatically detect the specific driving behavior; then, the daily and long-term driving behaviors of the driver are comprehensively evaluated by combining two modes of driving performance scores and driving level scores, and the comprehensive and practical effects are achieved, so that the driver can be better guided to evolve towards efficient driving and safe driving; finally, the driving data and the traffic management department can be networked to optimize traffic management and safety.
In the invention, a space-time characteristic fusion strategy is introduced, the detection efficiency can be further improved through embedded driving context information, and the daily and long-term bad driving habits of a driver can be improved by applying a self-defined driving behavior evaluation criterion. Compared with the conventional method in which the characteristic strategy is manually analyzed and selected, the method can more efficiently identify the spatiotemporal characteristics in the data and measure the importance of the spatiotemporal characteristics, can detect and feed back the driving behavior through a portable carrier, namely a smart phone, and greatly improves the application value of the method.
Drawings
FIG. 1 is a schematic view of a vehicle coordinate system in data preprocessing according to the present invention. Wherein A1 is a right side view, and A2 is a front view.
FIG. 2 is a flow chart of the method for detecting the driving behavior of the vehicle with the fusion of the space-time characteristics.
FIG. 3 is an architecture diagram of a spatiotemporal feature fusion network used in the present invention.
Fig. 4 is a structural diagram and a schematic workflow diagram of a driving behavior detection apparatus according to the present invention.
FIG. 5 shows the test results of the inventive vehicle driving behavior detection method on the UAH-Driveset data set.
Detailed Description
The invention is further illustrated by the following specific examples and the accompanying drawings.
The target vehicle coordinate system in the present invention is shown in fig. 1. The smart phone can be horizontally or vertically placed in the vehicle, and the original data coordinate systems of different mobile phones are different, so that the coordinate systems of the acquired data are not uniform. In order to unify the coordinate system, after the raw data in the sensor is acquired, the coordinate system of the raw data needs to be converted into a target vehicle coordinate system in the preprocessing operation, so that the data can meet the input requirement of the neural network. As shown in fig. 1, a1 is a right side view and a2 is a front view. The operation comprises the steps of off-line training and on-line detection, and data can be correctly processed through the converted coordinate system to detect and grade the driving behavior.
The invention provides a vehicle driving behavior detection method based on space-time feature fusion. The specific flow is shown in fig. 2, and specifically includes 6 key steps of data preprocessing L1, spatio-temporal feature extraction L2, driving behavior classification L3, real-time data processing S1, driving behavior detection S2, and driving behavior evaluation S3. The method can detect 7 driving behaviors: sudden braking, quick lane change, continuous lane change, quick left turn, quick right turn, quick turn around and normal driving. The invention introduces a space-time characteristic fusion strategy, can further improve the detection efficiency through the embedded driving context information, and can improve the daily and long-term bad driving habits of the driver by applying the self-defined driving behavior evaluation criterion. Compared with the conventional method in which the characteristic strategy is manually analyzed and selected, the method can more efficiently identify the spatiotemporal characteristics in the data and measure the importance of the spatiotemporal characteristics, can detect and feed back the driving behavior through a portable carrier, namely a smart phone, and greatly improves the application value of the method.
In an off-line training stage, firstly, preprocessing operations such as coordinate conversion, up-sampling (aiming at low sampling rate modal data), wavelet transformation denoising, maximum and minimum normalization, sliding window division and the like are carried out on multi-mode sensing off-line data acquired by a smart phone, wherein the multi-mode sensing off-line data comprises vehicle acceleration and sensing data such as a gyroscope, GPS data, a magnetic sensing meter, road types, weather types and the like; then, respectively carrying out feature extraction modeling on the spatial and temporal relationships of data in different modes through CNN and LSTM, and respectively carrying out weight measurement on the spatial features and the temporal features through a self-attention network; finally, classifying the space-time characteristics by using a fully-connected network layer and a Softmax function to obtain specific driving behavior classification; in the on-line detection stage, firstly, the dynamic data of the vehicle, including vehicle acceleration, direction, a magnetic sensor, GPS data and driving context information (weather type, road type, map information and the like), are acquired in real time through the application of the smart phone APP, and the real-time multi-mode data are preprocessed; then, detecting specific driving behavior classes by using the driving behavior classification model trained offline; and finally, calculating a driving level integral through a driving evaluation criterion, and warning the driver of bad driving habits.
The spatio-temporal feature fusion network architecture provided by the invention comprises a multi-modal input layer, an attention-based CNN fusion sub-network layer, an attention-based LSTM fusion sub-network layer and a full connection output layer, and is shown in figure 3.
Generating a multi-modal driving sequence through a multi-modal input layer, wherein the input of the multi-modal input layer is four modal information of acceleration, azimuth, a magnetic sensor and GPS, the duration ws of an input window is 5s, a time step ts is 0.5s, an input window comprises a time step T which is 10, and the sampling rate f of a data set hz 100, sequence of window inputsLength s w Is 500;
extracting local spatial features among different modes through a CNN fusion subnet based on attention, specifically, a CNN convolution kernel is 1 × 3, the number of channels is 64, a Dropout layer with an additional value of 0.2, a RELU active layer and a maximum pooling layer with a kernel of1 × 3 are added after each convolution layer, and the spatial features are subjected to importance measurement and fusion by using self-attention;
searching a long-term relationship between time periods through an attention-based LSTM fused subnet, specifically, receiving fused spatial features successively according to time steps, further extracting a time relationship in the features by using the LSTM, wherein the hidden state of the LSTM is 64, the optimal time step is 15, and weighting and fusing the hidden states of the LSTM output at all the time steps by using self-attention;
the driving behavior probability is calculated by embedding driving context information into a full connection layer, specifically, the driving context information (weather, road and map data) is embedded into fusion space-time characteristics as embedded information, the information is input into a neural network through one-hot coding instead of numerical value input, then the driving behavior classification is calculated for the fusion space-time characteristics through the full connection layer and a Softmax function, namely, the one-hot vectors of 100 full connection neurons and 2 driving contexts are combined, and 6 classifications are output. The batch processing size of the network training is 32, the iteration times are 500, 5-fold cross validation is applied, and the learning rate is 1 multiplied by 10 -4 。
Fig. 4 is a block diagram and a workflow diagram of a driving behavior detection apparatus according to an embodiment of the present invention, which includes a smart phone, a mobile phone software program, a computer memory, and a computer processor. The vehicle driving behavior detection method with the fusion of the space-time characteristics and the online feedback in the embodiment of the invention can be realized by respectively executing a computer software program by a computer processor and executing a mobile phone software program by a smart phone. As shown in fig. 4, the apparatus includes 4 processes;
a process P1, transferring the data multimode data collected by the smart phone to a computer storage device for model training and updating;
in the process P2, the computer processor reads data from the memory and obtains a driving behavior detection model through a model training algorithm;
the process P3 uploads the trained detection model to the smart phone irregularly, and model iteration is performed continuously, so that the model detection accuracy is improved;
the process P4 detects a specific driving behavior in real time by executing a mobile phone software program, and prompts a driver's bad driving habits according to an evaluation criterion. Through the set of complete off-line training and on-line detection equipment, the detection efficiency and the program execution flexibility can be obviously improved, and the practical value of application is further improved.
To investigate the necessity of each component, we excluded or replaced the components separately to observe the variation in model performance. FIG. 5 shows the results of tests on the UAH-Driveset data set with evaluation indices MacroF1-score (mF1), D1-D6 for 6 different drivers, and models 1-Model6 respectively representing the exclusion or replacement of modules in the neural network, including single-modality input sequences (UIS), Convolutional Neural Network (CNN), self-attention (SA), long-short term memory network (LSTM), multi-head attention (MHA), and Embedded Context Information (ECI). The result shows that each module in the method provided by the invention has an optimization effect on the final test result.
In the invention, the smart phone device for acquiring the multi-modal sensing data can be replaced by any embedded device.
In the invention, the modal types of the sensing data, such as acceleration, azimuth, magnetic sensor, GPS, road type, weather type, map context information and the like, can be selectively increased or decreased.
In the invention, the CNN network for extracting the spatial features may be replaced by other types of convolutional neural networks such as respet, inclusion, Densenet, and the like.
In the present invention, the LSTM network for extracting temporal features may be replaced with other types of recurrent neural networks such as RNNs, GRUs, and the like.
In the present invention, the score setting for the driving performance score and the driving level integral may be modified as needed, or only the driving performance score or the driving level integral may be applied.
The type and amount of driving behavior output in the present invention may be redefined based on the labels of the training data.
Some of the custom parameters involved in the present invention can be modified.

Claims (6)

1. A vehicle driving behavior detection method based on space-time feature fusion is characterized by comprising an off-line training stage and an on-line training stage; the off-line training stage comprises data preprocessing, space-time feature extraction and driving classification; the trained model is used for detecting driving behaviors in the multi-modal sensing information in the sliding window; the on-line detection stage comprises real-time data processing, driving behavior detection and driving behavior evaluation; loading the offline trained model through a software program of the smart phone to realize real-time driving behavior detection and driving behavior evaluation; the method comprises the following specific steps:
(1) data pre-processing
Collecting vehicle multimodal sensing offline data by a smartphone, the data comprising: vehicle acceleration and gyroscope, GPS data, magnetic sensor, road type, weather type sensing data, and preprocessing the data, including coordinate conversion, up-sampling, wavelet transformation denoising, maximum and minimum normalization, and sliding window division;
(2) spatio-temporal feature extraction
Respectively extracting the characteristics of the spatial and temporal relationships of different modal data through CNN and LSTM, and respectively weighing the spatial characteristics and the temporal characteristics through a self-attention network;
(3) classifying driving behavior
Classifying the space-time characteristics by using a fully-connected network layer and a Softmax function to obtain a specific driving behavior category;
(4) real-time data processing
Obtain the power data of vehicle in real time by smart mobile phone APP application, include: vehicle acceleration, direction, magnetic sensor, GPS data, driving context information, and pre-processing real-time multi-modal data;
(5) detecting driving behavior
Detecting specific driving behavior classes by using a driving behavior classification model trained offline;
(6) evaluation of driving behavior
Evaluating the driving behavior by adopting two evaluation rules; firstly, evaluating the driving performance score for standardizing the daily driving performance of a driver; second, the driving level integral evaluation is used for improving the driving habit of the driver;
wherein, the deduction formula of the driving performance score is as follows:
score＝100-count ano (1)
wherein, count ano Counting 1 time for abnormal driving behaviors of 5 times or more within 10 kilometers, and performing voice warning; the abnormal driving rows are divided into six types, which are respectively as follows: sudden braking, quick lane change, quick continuous lane change, quick left turn, quick right turn and quick turn around.
2. The spatiotemporal feature fusion-based vehicle driving behavior detection method according to claim 1, wherein the specific flow of the data preprocessing in step (1) is as follows:
first, consider that the raw signal is obtained by multiple smartphone sensors, where different sensors have different frequencies; therefore, the low frequency signal is up-sampled and specifically represented by linear interpolation filtering:
Figure FDA0003639074790000021
where os (-) and s (-) represent the up-sampled and original signals, i 2 And i 1 Respectively up-sampling the rear position and the front position of the position i;
then, decomposing the sampled signals through wavelet transformation to obtain a smoother sequence so as to improve the generalization capability of the model; the wavelet transform operation is represented as:
Figure FDA0003639074790000022
wherein, N, l m And N w Respectively representing the length of an original sequence, the number of decomposed layers and the length of a decomposition filter, and finally obtaining a sequence after denoising through inverse wavelet transform;
and, with max-min normalization, all variables are mapped to the range [0,1], denoted as:
Figure FDA0003639074790000023
wherein, s', s min And s max Respectively representing the minimum value and the maximum value of the original variable to be normalized, the normalized variable and the original variable;
finally, the preprocessed sequence is divided into a plurality of input sequences, and the input sequences are input into the network through a sliding window strategy, wherein the input sequences are expressed as follows:
Figure FDA0003639074790000024
wherein ts, ws and T are the duration of a single time step, the duration of a sliding window and the number of time steps, s, respectively w And f hz Respectively the total sequence length of a single sliding window and the sampling rate of the data set.
3. The method for detecting the driving behavior of the vehicle based on the spatiotemporal feature fusion as claimed in claim 2, wherein the spatiotemporal feature extraction in the step (2) comprises the following specific steps:
first, spatial features in driving behavior data are extracted through a CNN network, and are expressed as:
Figure FDA0003639074790000025
Figure FDA0003639074790000026
Figure FDA0003639074790000027
wherein v is tm Is c tm By means of the implicit representation of the full connection layer calculation,
Figure FDA0003639074790000028
and
Figure FDA0003639074790000029
are network parameters that are automatically learned in the tth time step in the CNN network,
Figure FDA00036390747900000210
is a normalized weight calculated by the Softmax function,
Figure FDA00036390747900000211
is a unified representation of all modalities after attention calculation;
next, the temporal features in the driving behavior data are extracted through the LSTM network, and are expressed as:
Figure FDA0003639074790000031
Figure FDA0003639074790000032
Figure FDA0003639074790000033
Figure FDA0003639074790000034
Figure FDA0003639074790000035
wherein i t 、f t 、o t 、
Figure FDA0003639074790000036
And C t An input gate, a forgetting gate, an output gate, a candidate memory unit and a memory unit of the LSTM respectively;
Figure FDA0003639074790000037
and h t-1 The hidden representation output of the t-th time step of the CNN network and the hidden state of the LSTM network at the t-1 moment are respectively; w and b respectively represent the network parameters automatically learned by the gate control unit in the LSTM network; the extracted temporal features are further weighted by the full connectivity layer and the Softmax function for the importance of different time steps, expressed as:
k t ＝tanh(W 2 h t +b 2 )， (14)
Figure FDA0003639074790000038
δ＝∑ t β t h t ， (16)
wherein k is t 、β t And delta respectively represents the hidden representation of the LSTM hidden state obtained after the calculation of the full connection layer, the normalized weight of the time characteristic and the unified representation of all time steps after the attention calculation; h is t Is the input of the t time step of the full connection layer, w 2 、W 2 And b 2 Respectively, representing automatically learned network parameters in a fully connected network.
4. The method for detecting the driving behavior of the vehicle based on the spatiotemporal feature fusion as claimed in claim 3, wherein the specific process for classifying the driving behavior in the step (3) is as follows:
embedding driving context information into a full connection layer, classifying driving behaviors, and expressing as follows:
e r ＝onehot(v)×W e ， (17)
Figure FDA0003639074790000039
Figure FDA00036390747900000310
wherein e is r 、
Figure FDA00036390747900000311
And r behavior Respectively representing embedded driving context information, a driving behavior classification calculated by a Softmax function and a specific driving behavior class; onehot (v) and f s Respectively representing the independent variables of the driving context information v and the hidden representation obtained by the previous layer by using the full-connection calculation again for the output delta of the attention network; argmax (·) is the driving behavior with the highest probability in solving the predicted driving behavior category; w is a group of e 、W y And b y Network parameters that represent classification layer network auto-learning.
5. The spatio-temporal feature fusion-based vehicle driving behavior detection method according to claim 4, characterized in that the driving behavior is evaluated in step (6), wherein:
the driving performance is divided into 5 grades, the initial value is 100 grades, and the driving performance grades are respectively divided into a table 1; when the driving behavior is normal driving and the score is lower than 100 minutes, adding 1 score every 10 kilometers; when the driving performance is graded to be 2, namely the driving consciousness is extremely poor, the driver is not recommended to drive on the highway section; when the driving performance is graded to be 1, namely the driving consciousness is bad, the driver is not advised to drive on the urban road section;
the driving level integral is divided into 7 grades, the specific grade division is shown in a table 2, the initial score is 750, and the grade is 1; when the driving performance grade is 4 or above, adding 1 integral corresponding to the integral grade of the current driving level, namely the 1 st column in the table 2, and adding 1 integral when the driver normally drives for a certain kilometer number, namely the 3 rd column in the table 2; only when 5 times or more of abnormal driving behaviors are accumulated in every 100 kilometers of driving, 5 points are deducted;
TABLE 1 Driving Performance Classification
Performance level classification Score segment Description of the invention 5 100 Excellent driving awareness 4 [90,99] Good driving consciousness 3 [80,89] Poor driving consciousness 2 [60,79] Poor consciousness of driving 1 <＝59 Bad driving consciousness
TABLE 2 Driving level score rating
Driving horizon rating Score segment Number of kilometers required for bonus 7 >＝1800 40 6 [1600,1799] 35 5 [1400,1599] 30 4 [1200,1399] 25 3 [1000,1199] 20 2 [800,999] 15 1 <＝799 10
。
6. A detection device based on the detection method for vehicle driving behavior based on spatio-temporal feature fusion of claims 1-5, characterized by comprising a smart phone, a mobile phone software program, a computer memory, a computer processor; respectively executing a computer software program and a mobile phone software program through a computer processor to realize the vehicle driving behavior detection and online feedback of the time-space characteristic fusion; the specific operation flow is as follows:
a process P1, transferring the multimodal data collected by the smart phone to a computer storage device for model training and updating;
in the process P2, the computer processor reads data from the memory and obtains a driving behavior detection model through a model training algorithm;
the process P3 uploads the trained detection model to the smart phone irregularly, and model iteration is performed continuously, so that the model detection accuracy is improved;
the process P4 detects a specific driving behavior in real time by executing a mobile phone software program, and prompts a driver's bad driving habits according to an evaluation criterion.
CN202210547632.4A 2022-05-11 2022-05-11 Vehicle driving behavior detection method based on space-time feature fusion Pending CN114926825A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210547632.4A CN114926825A (en) 2022-05-11 2022-05-11 Vehicle driving behavior detection method based on space-time feature fusion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210547632.4A CN114926825A (en) 2022-05-11 2022-05-11 Vehicle driving behavior detection method based on space-time feature fusion

Publications (1)

Publication Number Publication Date
CN114926825A true CN114926825A (en) 2022-08-19

Family

ID=82808801

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210547632.4A Pending CN114926825A (en) 2022-05-11 2022-05-11 Vehicle driving behavior detection method based on space-time feature fusion

Country Status (1)

Country Link
CN (1) CN114926825A (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116343177A (en) * 2023-03-02 2023-06-27 安庆梁葛业网络科技有限公司 Method and system for monitoring intelligent traffic abnormal driving behavior based on data processing
CN116991157A (en) * 2023-04-14 2023-11-03 北京百度网讯科技有限公司 Automatic driving model with human expert driving capability, training method and vehicle
CN117056799A (en) * 2023-08-03 2023-11-14 广东省机场管理集团有限公司工程建设指挥部 Processing method, device, equipment and medium for vehicle sensor data
CN117315428A (en) * 2023-10-30 2023-12-29 燕山大学 Cross-modal feature alignment and fusion crowd counting system and method
CN117724137A (en) * 2023-11-21 2024-03-19 江苏北斗星通汽车电子有限公司 Automobile accident automatic detection system and method based on multi-mode sensor

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116343177A (en) * 2023-03-02 2023-06-27 安庆梁葛业网络科技有限公司 Method and system for monitoring intelligent traffic abnormal driving behavior based on data processing
CN116991157A (en) * 2023-04-14 2023-11-03 北京百度网讯科技有限公司 Automatic driving model with human expert driving capability, training method and vehicle
CN117056799A (en) * 2023-08-03 2023-11-14 广东省机场管理集团有限公司工程建设指挥部 Processing method, device, equipment and medium for vehicle sensor data
CN117056799B (en) * 2023-08-03 2024-03-26 广东省机场管理集团有限公司工程建设指挥部 Processing method, device, equipment and medium for vehicle sensor data
CN117315428A (en) * 2023-10-30 2023-12-29 燕山大学 Cross-modal feature alignment and fusion crowd counting system and method
CN117315428B (en) * 2023-10-30 2024-04-05 燕山大学 Cross-modal feature alignment and fusion crowd counting system and method
CN117724137A (en) * 2023-11-21 2024-03-19 江苏北斗星通汽车电子有限公司 Automobile accident automatic detection system and method based on multi-mode sensor

Similar Documents

Publication Publication Date Title
CN114926825A (en) Vehicle driving behavior detection method based on space-time feature fusion
CN110263846B (en) Fault diagnosis method based on fault data deep mining and learning
CN111160139B (en) Electrocardiosignal processing method and device and terminal equipment
CN110097037B (en) Intelligent monitoring method and device, storage medium and electronic equipment
CN111859010B (en) Semi-supervised audio event identification method based on depth mutual information maximization
CN108694408B (en) Driving behavior recognition method based on deep sparse filtering convolutional neural network
CN104749334A (en) Mode-recognition-based design method for biological abnormal water quality evaluation system
CN116010885A (en) Method and system for detecting abnormal space-time data of vehicle under long-sequence condition
CN113848417A (en) Rail transit power supply equipment fault prediction method and device
CN112699793A (en) Fatigue driving detection optimization identification method based on random forest
CN112529638B (en) Service demand dynamic prediction method and system based on user classification and deep learning
Ma et al. Real-time detection of abnormal driving behavior based on long short-term memory network and regression residuals
CN112504682A (en) Chassis engine fault diagnosis method and system based on particle swarm optimization algorithm
CN117421684A (en) Abnormal data monitoring and analyzing method based on data mining and neural network
CN110263836B (en) Bad driving state identification method based on multi-feature convolutional neural network
CN114926299A (en) Prediction method for predicting vehicle accident risk based on big data analysis
CN111476102A (en) Safety protection method, central control equipment and computer storage medium
CN113707175A (en) Acoustic event detection system based on feature decomposition classifier and self-adaptive post-processing
CN114331214A (en) Domain-adaptive bearing voiceprint fault diagnosis method and system based on reinforcement learning
CN111968003B (en) Crop disease prediction method based on crop ontology concept response
CN116714437B (en) Hydrogen fuel cell automobile safety monitoring system and monitoring method based on big data
CN116946183A (en) Commercial vehicle driving behavior prediction method considering driving capability and vehicle equipment
CN116955945A (en) Ship abnormality detection method based on transducer_LSTM
CN116594057A (en) Earthquake early warning method and device based on deep learning and edge calculation
CN116738330A (en) Semi-supervision domain self-adaptive electroencephalogram signal classification method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination