CN115902878A - Millimeter wave radar human behavior recognition method - Google Patents
Millimeter wave radar human behavior recognition method Download PDFInfo
- Publication number
- CN115902878A CN115902878A CN202211461976.XA CN202211461976A CN115902878A CN 115902878 A CN115902878 A CN 115902878A CN 202211461976 A CN202211461976 A CN 202211461976A CN 115902878 A CN115902878 A CN 115902878A
- Authority
- CN
- China
- Prior art keywords
- mdm
- human body
- time
- data set
- millimeter wave
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention relates to the field of human behavior recognition, and discloses a millimeter wave radar human behavior recognition method, which comprises the following steps: the first step is as follows: preprocessing the collected radar human body echo signals to generate an MDM (multiple-driven mezzanine) diagram, and extractingThe second step is that: by short-time Fourier transform pairsProcessing to generate MDM, and constructing a human body behavior data set by using the generated MDM, wherein the human body behavior data set comprises a training data set Otrain and a test data set Otest; the third step: the millimeter wave radar human body behavior recognition method adopts the column-based MDM image division instead of the block-based MDM image division, and follows the characteristic that an MDM image reflects the time-varying change of micro Doppler components. And the picture position code after slicing is given through the 1-dimensional position code, and the space-time characteristic of the time sequence is fully extracted.
Description
Technical Field
The invention relates to the field of human behavior recognition, in particular to a millimeter wave radar human behavior recognition method.
Background
Human behavior recognition (HAR) has very wide application prospects in many fields such as safety precaution, anomaly detection and the like, and the fields need to quickly and accurately recognize human behaviors. Meanwhile, along with the popularization and development of artificial intelligence, the flexibility and accuracy of the technology are further improved. Camera-based HARs suffer from privacy leakage problems and are not suitable for dark or other low-light scenes. The HAR based on radar draws wide attention due to the advantages of all weather, strong privacy, adaptability to any illumination condition and the like all the day. In particular, frequency Modulated Continuous Wave (FMCW) radar has the advantages of high resolution, high detection accuracy, small size, low cost, and the like, and is more prominent in HAR.
The method has wide application in radar by utilizing Micro Doppler Features (MDF) generated by target motion for target identification. In the HAR based on the radar micro-doppler characteristic, the characteristic difference provided by doppler generated by the movement of the trunk of a human body is very small, and the characteristic with higher differentiation mainly comes from the micro-doppler generated by the swinging of four limbs, so the MDF is the basis for identifying different behaviors of the human body. MDF is the characteristic of micro Doppler frequency shift generated by human body movement along with time, and can be observed in a joint time-frequency domain. How to extract efficient MDFs is a core issue of HAR tasks. In the early stages, researchers need to rely on a priori knowledge to extract MDF manually, or extract MDF through Empirical Mode Decomposition (EMD), in combination with traditional Machine Learning (ML) to achieve HAR. However, the features extracted in this way are not only time-consuming and labor-consuming, but also are only suitable for a specific task, and the generalization capability of the network is limited.
The rapid development of Deep Learning (DL) in recent years has brought new research ideas to radar-based HARs. Compared with the traditional ML, the DL does not depend on manual experience to extract features, can automatically learn the inherent features of data, and has excellent performance and strong generalization capability. The classical DL model Convolutional Neural Network (CNN) and its various deep variant forms have been widely used in HAR tasks. The reference rate applies deep CNN to HAR first and achieves good classification effect. And different features are extracted by using the multi-dimensional CNN in the HAR for fusion so as to improve the identification precision. In addition, the shallow CNN network can reduce the complexity of the network and improve the operation efficiency of the network. If only two layers of CNNs are designed to detect human body movement, the number of network parameters is small, the requirement on calculation force is not high, but the recognition effect on certain behaviors is poor. The above mentioned model regards the representation of human behavior as a visual image from which spatially correlated features between pixels are extracted through a deep network. However, an original human body echo signal acquired by a radar is time sequence data, an MDF has spatial and temporal relevance, a deep multidimensional CNN ignores a time dependency relationship between time sequences, and recognition accuracy is limited, so that a millimeter wave radar human body behavior recognition method is provided.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides a millimeter wave radar human body behavior identification method, which solves the problems.
(II) technical scheme
In order to achieve the above purpose, the invention provides the following technical scheme: a millimeter wave radar human behavior recognition method comprises the following steps:
the first step is as follows: for collected radar human body echo informationPreprocessing the MDM to generate an MDM graph, and extracting the MDM graph
The second step is that: by short-time Fourier transform pairsProcessing to generate MDM, and constructing a human body behavior data set by using the generated MDM, wherein the human body behavior data set comprises a training data set Otrain and a test data set Otest;
the third step: and extracting MDF characteristics from the constructed MDM data set by using a VIT network model based on time slice segmentation to realize HAR.
Preferably, the pretreatment in the first step comprises the steps of:
s1: the radar receives signals reflected by a human body and the surrounding environment, and carries out frequency mixing with the transmitted signals, and the output result is an intermediate frequency signal;
s2: performing distance compression and angle compression by using a 2D FFT (fast Fourier transform), concentrating the energy of each frame of signal on a position and angle grid of a target, and generating a plurality of two-dimensional matrixes with slow time dimensions
S3: and (3) background clutter suppression, wherein most strong static clutter is removed by adopting a phasor mean value cancellation algorithm, and human motion signals cannot be influenced. The two-dimensional matrix sequence subjected to clutter removal is
S4: and (3) locking the human target motion grid by adopting a moving average-order statistical constant false alarm rate (MAOS-CFAR) algorithm. Extracting the vector of the grid where the target is located along the slow time dimension
Preferably, the intermediate frequency signal is:
in the formula (f) c For the radar carrier frequency, B is the signal bandwidth, T is the signal duration, R is the distance from the radar to the target, V is the radial velocity of the target, and c represents the speed of light. N =0,1,2., N-1 is the number of sample points on a single chirp, L =0,1,2., L-1 is the number of chirps in a single frame, and t represents the current t-th frame.
Preferably, the third step comprises the following specific steps:
s1: the MDM is divided chronologically into a number of tiles of the same time step, with no overlap between each tile. Each image block can be regarded as a one-dimensional micro Doppler time sequence, the time sequences are vectorized respectively, the position relation of adjacent time sequences is extracted in the time dimension, and the complete time information of MDM is reserved;
s2: firstly, generating a characteristic sequence of each slice sequence by using one-dimensional CNN, and then adding position information into the slice sequences by adopting 1-dimensional position coding together with token sequences for classification;
s3: transmitting the characteristic sequence into a normalization module to generate three trainable variables Q, K and V, and transmitting the three parameters into a multi-head attention machine to train;
s4: training VIT models of all training samples according to S1-S3, using the trained models for the test sample data, and testing the final recognition effect of the HAR.
Preferably, the specific steps of S3 are as follows:
MultiheadAttention(Q,K,V)=Concat(head1...head i )W O ;
head i =Attention(Q,K,V);
multi-headed attention maps Q and K to multiple different subspaces of the original high-dimensional space to compute similarity, which requires Q-K-v scaling. This scaling step is done by a number of linear modules at the input of the encoding block. The number of linear modules is determined by the number of heads. And sending the proportion groups of Q-K-V into a plurality of proportion point attention modules, and respectively extracting the time sequence characteristics of the Doppler sequence. And finally, synthesizing feature information output by different attention networks by using the Concat. The essence of multi-head attention is to find the characteristic correlation between doppler sequences of different scales by fusing the attention information from different distribution subspaces. This spatial decomposition and resynthesis can reduce the vector dimension each time the attention head is computed, preventing overfitting to some extent.
(III) advantageous effects
Compared with the prior art, the invention provides a millimeter wave radar human behavior identification method, which has the following beneficial effects:
1. according to the millimeter wave radar human body behavior identification method, the MDM image division is carried out according to the columns instead of the blocks, and the characteristic that the MDM image reflects the time-varying change of the micro Doppler component is followed. The picture slice position coding is given through the 1-dimensional position coding, and the time sequence space-time characteristics are fully extracted.
2. According to the millimeter wave radar human behavior recognition method, a multi-head attention mechanism is adopted to focus more important time characteristics, effective utilization of all time is guaranteed, and recognition accuracy of human behaviors is effectively improved.
Drawings
FIG. 1 is a schematic flow diagram of the process;
FIG. 2 is a diagram of Vision Transformer network model extraction MDF;
FIG. 3 is a schematic diagram of a computing process of the multi-head self-attention mechanism;
FIG. 4 is a micro-Doppler spectrum and a characteristic thermodynamic diagram while walking;
FIG. 5 is a micro Doppler spectrum and a characteristic thermodynamic diagram during running;
FIG. 6 is a micro-Doppler spectrum and a characteristic thermodynamic diagram during squatting and standing;
FIG. 7 is a bow micro-Doppler spectrum and a characteristic thermodynamic diagram;
FIG. 8 is a micro-Doppler spectrum and a characteristic thermodynamic diagram during a turn;
FIG. 9 is a graph showing the comparison of performance of different models;
fig. 10 is a diagram illustrating comparison of recognition effects of block-wise division and column-wise division.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The scheme combines the micro Doppler characteristic and a Vision Transformer (VIT) network model in the frequency modulation continuous wave radar to provide an efficient HAR method, and a specific flow chart is shown in figure 1. By preprocessing the radar original human body signal, a clear micro Doppler spectrogram (MDM) capable of representing MDF is generated. The MDM is used as the input of a VIT model, the MDM is divided into sequences with the same time step, 1D pos-embedding is adopted to add position codes and extract the spatial characteristics of time sequences, the weight distribution mechanism of Multi-head-Attention can focus more important time characteristics, effective utilization of all time is guaranteed, the efficiency of network information processing is improved, and the extracted MDF is input into an MLP classifier to complete recognition of human body behaviors. The scheme has the advantages of stable network performance, high mobility, high identification accuracy rate and the like, and can meet the requirements of accurate and real-time HAR.
The HAR method of the scheme comprises the following steps:
step 1, preprocessing the collected radar human body echo signals to generate an MDM image. The pre-processing procedure includes 2D FFT, background clutter suppression, and Constant False Alarm Rate (CFAR) detection.
Step 1.1, the radar receives signals reflected by a human body and the surrounding environment, and carries out frequency mixing with the transmitted signals, and the output result is an intermediate frequency signal. Considering the case where a radar transmits a multi-frame signal, the intermediate frequency signal can be expressed as:
in the formula, f c For radar carrier frequency, B is signal bandwidth, T is signal duration, R is distance between radar and target, V is radial velocity of target, and c represents speed of light. N =0,1,2.. N-1 is the number of sample points on a single chirp, L =0,1,2.. L-1 is the number of chirp in a single frame, and t denotes the current t-th frame.
Step 1.2, using 2D FFT to carry out distance compression and angle compression, concentrating the energy of each frame signal on a position and angle grid of a target, and generating a plurality of two-dimensional matrixes with slow time dimension
And step 1.3, in order to avoid covering the human motion signals in the clutter environment with strong interference such as the background and the like, removing most of strong stationary clutter by adopting a phasor mean value cancellation algorithm without influencing the human motion signals. The two-dimensional matrix sequence subjected to clutter removal is
And step 1.4, in order to lock the two-dimensional grid where the target is located, a moving grid of the human target is locked by adopting a moving average-order statistical constant false alarm rate (MAOS-CFAR) algorithm. Extracting the vector of the grid where the target is located along the slow time dimension
Step 2, through short-time Fourier transform pairAnd (5) processing to generate the MDM. And constructing a human body behavior data set by using the generated MDM, wherein the human body behavior data set comprises a training data set Otrain and a testing data set Otest.
And 3, extracting MDF characteristics from the MDM data set constructed in the step 2 by using a VIT network model to realize HAR, wherein the whole process is shown in FIG. 2.
Step 3.1, the MDM is divided into a plurality of blocks with the same time step according to the time sequence, and each block has no overlap. Each image block can be regarded as a one-dimensional micro Doppler time sequence, the time sequences are vectorized respectively, the position relation of adjacent time sequences is extracted in the time dimension, and the complete time information of the MDM is reserved.
And 3.2, generating a characteristic sequence of each slice sequence by using the one-dimensional CNN, and adding position information into the slice sequences by adopting 1-dimensional position coding together with token sequences for classification.
And 3.3, transmitting the characteristic sequence into a normalization module to generate three trainable variables Q, K and V, and transmitting the three parameters into a multi-head attention mechanism for training, wherein the calculation process is as shown in FIG. 3.
Multi-head-Attention(Q,K,V)=Concat(head1...head i )W O ;
head i =Attention(Q,K,V);
Multi-headed attention maps Q and K to multiple different subspaces of the original high-dimensional space to compute similarity, which requires Q-K-V scaling. This scaling step is done by a number of linear modules at the input of the encoding block. The number of linear modules is determined by the number of heads. And sending the proportion groups of Q-K-V into a plurality of proportion point attention modules, and respectively extracting the time sequence characteristics of the Doppler sequence. And finally, synthesizing feature information output by different attention networks by using the Concat. The nature of multi-head attention is to find the characteristic correlation between doppler sequences of different scales by fusing the attention information from different distribution subspaces. This spatial decomposition and resynthesis can reduce the vector dimension at each head of attention calculation, preventing overfitting to some extent.
And 3.4, training the VIT model by all training samples of Otrain according to the steps 3.1-3.4, using the trained model for the Otest sample data, and testing the final recognition effect of the HAR.
The method is explained in detail by combining experiments, and the experiments apply the method to actual HAR to verify the performance of the method. The specific implementation details are as follows:
the experiments utilized TI AWR1843 radar to collect human behavior data. The radar parameters are set as follows: the carrier frequency is 77GHz, sawtooth frequency modulation continuous waves are transmitted, the number of sampling points is 128, the Doppler resolution is 0.05m/s, and the single behavior acquisition time is 5s. In order to achieve a sufficient detection area, the height of the radar is 1.5m, and the distance between an experimenter and the radar is 3-4m. Five types of common human body behaviors are designed through experiments, namely (1) walking, (2) running, (3) squatting and standing, (4) bending and turning. To generalize the data, 7 men and 3 women were selected from the experiment to participate in the data collection, and their heights, weights and ages varied. Each action was repeatedly performed 20 times by 10 experimenters for a total of 1000 (5 × 10 × 20) groups of human body behavior data. When experimental data are collected, only aiming at a single experimenter scene, strict constraint is not imposed on the normative of behaviors, and the method can be executed according to personal habits so as to ensure the diversity of data sets. The radar raw data processing procedure is implemented in MATLAB v 2017. The output size of the 2D FFT is 134 × 256. One frame of MDM is generated in 50ms, the total accumulation time of the MDM is 5s in 100 frames, and MDMs of five types of human behaviors are shown in figure 9. The results of fig. 4-8 show that different human behaviors have unique MDFs, which are the basis for human behavior recognition. The MDM is converted into a gray scale image, the size of the image is reshaped to be 112 x 112, original detail characteristics of the MDM are not lost after reshaping, and meanwhile the calculation amount of the CLA mixed multi-network model can be reduced. To ensure that there are enough samples in the dataset, data enhancement is also used, with the total amount of data ultimately generated being 4950, as 8: the ratio of 2 is divided into otalin and ottest (3465. VIT model was trained under the DL framework of pyrorch v1.11.0, python3.9 with network parameters set to: and the time step 112 is used for dividing the MDM into blocks with the same time step, the size of each block is 1 multiplied by 112, and the MDM input is completed after all the time steps. The training model adopts an Adam optimizer, the learning rate and the number of training rounds, and the number of batch samples of each iteration is 0.001, 40 and 200 respectively. Meanwhile, a Dropout method is used for closing some neurons in the network in the experiment, and the problem that overfitting causes reduction of the generalization capability of the model is avoided.
The characteristic thermodynamic diagrams of fig. 4-8 demonstrate that the present method is capable of focusing attention to a region of micro-doppler representing motion of the limb. The method is compared and analyzed with a plurality of common HAR models, the experimental result is shown in figure 9, and it can be seen that the average accuracy of the method can reach more than 99%, and the method is the best of all models.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (5)
1. A millimeter wave radar human behavior recognition method is characterized by comprising the following steps:
the first step is as follows: preprocessing the collected radar human body echo signals to generate an MDM (multiple-driven mezzanine) diagram, and extracting
The second step is that: by short-time Fourier transform pairsProcessing to generate MDM, and constructing a human body behavior data set by using the generated MDM, wherein the human body behavior data set comprises a training data set Otrain and a testing data set Otest;
the third step: and extracting MDF characteristics from the constructed MDM data set by using a VIT network model based on time slice segmentation to realize HAR.
2. The millimeter wave radar human body behavior recognition method according to claim 1, characterized in that: the pretreatment in the first step comprises the steps of:
s1: the radar receives signals reflected by a human body and the surrounding environment, and mixes the signals with the transmitted signals, and the output result is an intermediate frequency signal;
s2: performing distance compression and angle compression by using 2D FFT (two-dimensional Fourier transform), concentrating the energy of each frame of signal on a position and angle grid of a target, and generating a plurality of two-dimensional matrixes with slow time dimensions
S3: and (3) background clutter suppression, wherein most strong static clutter is removed by adopting a phasor mean value cancellation algorithm, and human motion signals cannot be influenced. The two-dimensional matrix sequence subjected to clutter removal is
3. The millimeter wave radar human body behavior recognition method according to claim 2, characterized in that: the intermediate frequency signal is:
in the formula (f) c For radar carrier frequency, B is signal bandwidth, T is signal duration, R is distance between radar and target, V is radial velocity of target, and c represents speed of light. N =0,1,2., N-1 is the number of sample points on a single chirp, L =0,1,2., L-1 is the number of chirps in a single frame, and t represents the current t-th frame.
4. The millimeter wave radar human body behavior recognition method according to claim 1, characterized in that: the third step comprises the following specific steps:
s1: the MDM is divided into a plurality of image blocks with the same time step length according to the time sequence, each image block does not overlap, each image block can be regarded as a one-dimensional micro-Doppler time sequence, the time sequences are respectively vectorized, the position relation of adjacent time sequences is extracted in the time dimension, and the complete time information of the MDM is reserved;
s2: firstly, generating a characteristic sequence of each slice sequence by using one-dimensional CNN, and then adding position information into the slice sequences by adopting 1-dimensional position coding together with token sequences for classification;
s3: transmitting the characteristic sequence into a normalization module to generate three trainable variables Q, K and V, and transmitting the three parameters into a multi-head attention machine to train;
s4: training VIT models of all training samples according to S1-S3, using the trained models for the test sample data, and testing the final recognition effect of the HAR.
5. The millimeter wave radar human body behavior recognition method according to claim 4, characterized in that: the specific steps of S3 are as follows:
MultiheadAttention(Q,K,V)=Concat(headl,。。。,,head i )W O ;
headi=Attention(Q,K,V);
attention is to map Q and K into the same high-dimensional space to calculate similarity, and Multi-head attribute is to map Q and K into high-dimensional space (a) 1 ,a 2 ,a 3 ,a 4 ,a i ) The Multi-head orientation is to map the same Q, K and V to different subspaces of the original high-dimensional space for Attention calculation under the condition that the total quantity of parameters is kept unchanged, and then to merge Attention information in different subspaces in the last stepAnd (4) information.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211461976.XA CN115902878A (en) | 2022-11-21 | 2022-11-21 | Millimeter wave radar human behavior recognition method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202211461976.XA CN115902878A (en) | 2022-11-21 | 2022-11-21 | Millimeter wave radar human behavior recognition method |
Publications (1)
Publication Number | Publication Date |
---|---|
CN115902878A true CN115902878A (en) | 2023-04-04 |
Family
ID=86492957
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202211461976.XA Pending CN115902878A (en) | 2022-11-21 | 2022-11-21 | Millimeter wave radar human behavior recognition method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN115902878A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116824629A (en) * | 2023-06-02 | 2023-09-29 | 大连理工大学 | High-robustness gesture recognition method based on millimeter wave radar |
CN117031434A (en) * | 2023-10-08 | 2023-11-10 | 中国科学技术大学 | Real-time falling detection method based on millimeter wave radar |
-
2022
- 2022-11-21 CN CN202211461976.XA patent/CN115902878A/en active Pending
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN116824629A (en) * | 2023-06-02 | 2023-09-29 | 大连理工大学 | High-robustness gesture recognition method based on millimeter wave radar |
CN117031434A (en) * | 2023-10-08 | 2023-11-10 | 中国科学技术大学 | Real-time falling detection method based on millimeter wave radar |
CN117031434B (en) * | 2023-10-08 | 2024-02-20 | 中国科学技术大学 | Real-time falling detection method based on millimeter wave radar |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Wang et al. | TS-I3D based hand gesture recognition method with radar sensor | |
CN115902878A (en) | Millimeter wave radar human behavior recognition method | |
Liu et al. | Deep learning and recognition of radar jamming based on CNN | |
CN111476058B (en) | Gesture recognition method based on millimeter wave radar | |
CN112597815A (en) | Synthetic aperture radar image ship detection method based on Group-G0 model | |
CN113466852B (en) | Millimeter wave radar dynamic gesture recognition method applied to random interference scene | |
CN115343704A (en) | Gesture recognition method of FMCW millimeter wave radar based on multi-task learning | |
Yuan et al. | STransUNet: A siamese TransUNet-based remote sensing image change detection network | |
Wang et al. | Research on feature extraction and recognition method of underwater acoustic target based on deep convolutional network | |
Jin et al. | Interference-robust millimeter-wave radar-based dynamic hand gesture recognition using 2D CNN-transformer networks | |
Li et al. | Multi-spectral remote sensing images feature coverage classification based on improved convolutional neural network | |
Williams et al. | Maritime radar target detection using convolutional neural networks | |
Zhu et al. | Multi-task self-supervised learning for vehicle classification based on carrier-free UWB radars | |
Song et al. | High-accuracy gesture recognition using mm-wave radar based on convolutional block attention module | |
Li et al. | Advancing IR-UWB radar human activity recognition with Swin-transformers and supervised contrastive learning | |
Ditzel et al. | Genradar: Self-supervised probabilistic camera synthesis based on radar frequencies | |
CN112132104B (en) | ISAR ship target image domain enhancement identification method based on loop generation countermeasure network | |
Li et al. | Supervised contrastive learning for vehicle classification based on the IR-UWB radar | |
Yan et al. | A New Method of Video SAR Ground Moving Target Detection and Tracking Based on the Inter-frame Amplitude Temporal Curve | |
Wang et al. | Attention-based vision transformer for human activity classification using mmwave radar | |
CN113421281A (en) | Pedestrian micromotion part separation method based on segmentation theory | |
Guo et al. | Cascaded ASPP and attention mechanism-based DeepLabv3+ semantic segmentation model | |
Zhong et al. | Human motion recognition in small sample scenarios based on GaN and CNN models | |
Xiang et al. | TCM Model for improving track sequence classification in real scenarios with Multi-Feature Fusion and Transformer Block | |
CN116908806B (en) | Multi-dimensional feature target detection method based on self-supervised learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |