CN112587153B - End-to-end non-contact atrial fibrillation automatic detection system and method based on vPPG signal - Google Patents
End-to-end non-contact atrial fibrillation automatic detection system and method based on vPPG signal Download PDFInfo
- Publication number
- CN112587153B CN112587153B CN202011423595.3A CN202011423595A CN112587153B CN 112587153 B CN112587153 B CN 112587153B CN 202011423595 A CN202011423595 A CN 202011423595A CN 112587153 B CN112587153 B CN 112587153B
- Authority
- CN
- China
- Prior art keywords
- atrial fibrillation
- vppg
- layer
- network
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 206010003658 Atrial Fibrillation Diseases 0.000 title claims abstract description 114
- 238000001514 detection method Methods 0.000 title claims abstract description 73
- 238000000034 method Methods 0.000 title description 14
- 238000012549 training Methods 0.000 claims abstract description 29
- 230000015654 memory Effects 0.000 claims abstract description 27
- 230000009467 reduction Effects 0.000 claims abstract description 23
- 239000012634 fragment Substances 0.000 claims abstract description 20
- 238000000605 extraction Methods 0.000 claims abstract description 18
- 230000007246 mechanism Effects 0.000 claims abstract description 18
- 238000007781 pre-processing Methods 0.000 claims abstract description 10
- 238000013528 artificial neural network Methods 0.000 claims abstract description 8
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 25
- 238000012360 testing method Methods 0.000 claims description 8
- 230000004927 fusion Effects 0.000 claims description 7
- 230000004913 activation Effects 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 4
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000007906 compression Methods 0.000 claims description 3
- 230000006835 compression Effects 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 5
- 239000011159 matrix material Substances 0.000 description 8
- 230000001815 facial effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 238000013144 data compression Methods 0.000 description 3
- 238000003745 diagnosis Methods 0.000 description 3
- 230000001788 irregular Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000013186 photoplethysmography Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 239000008280 blood Substances 0.000 description 2
- 210000004369 blood Anatomy 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000008034 disappearance Effects 0.000 description 2
- 230000007787 long-term memory Effects 0.000 description 2
- 230000006403 short-term memory Effects 0.000 description 2
- 208000024172 Cardiovascular disease Diseases 0.000 description 1
- 206010019280 Heart failures Diseases 0.000 description 1
- 208000007536 Thrombosis Diseases 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 206010003119 arrhythmia Diseases 0.000 description 1
- 230000006793 arrhythmia Effects 0.000 description 1
- 210000001367 artery Anatomy 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008033 biological extinction Effects 0.000 description 1
- 230000017531 blood circulation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000009966 trimming Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
- A61B5/02416—Detecting, measuring or recording pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7225—Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Signal Processing (AREA)
- Physiology (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Cardiology (AREA)
- Power Engineering (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an end-to-end non-contact atrial fibrillation automatic detection system based on a vPPG signal, which comprises the following components: the data preprocessing module is used for recording face videos of users to be detected, removing the positions with large noise interference at the beginning and the end of the recorded videos, downsampling, and intercepting the downsampled videos into uniform length and size; the pulse wave extraction module is used for extracting a vPPG signal of the face video through the P3D convolutional neural network; the data noise reduction module is used for removing noise from vPPG data based on the neural network of the FCN-DN; the atrial fibrillation detection module is firstly used for training a model to enable the atrial fibrillation detection module to learn to divide atrial fibrillation fragments and non-atrial fibrillation fragments, and then inputting the vPPG signal fragments to be detected into the trained atrial fibrillation detection model, so that whether the vPPG signal to be detected contains the atrial fibrillation fragments or not is judged. The parallel network combining the long-short time memory network containing the attention mechanism and the convolution network enables the detection of the model to be comprehensive, high in precision and good in effect.
Description
Technical Field
The invention belongs to the technical field of medical data analysis, and particularly relates to an atrial fibrillation automatic detection system and method based on a P3D convolution and attention mechanism long-time memory network aiming at a face video.
Background
Atrial fibrillation is an arrhythmia caused by abnormal heart activity. Elderly patients over 80 years old have typical rates of atrial fibrillation between 10% and 17%. The number of patients suffering from atrial fibrillation in the whole world in 2020 has reached 3300 ten thousand. Later atrial fibrillation is usually accompanied by cardiovascular diseases such as thrombosis and heart failure. Therefore, it is important to early find and early prevent atrial fibrillation signals. Atrial fibrillation is clinically diagnosed by a physician analyzing a 12-lead ECG (electrocardiogram) signal, with the criteria for an electrocardiogram diagnosis being an irregular heart rate duration exceeding 30 seconds, accompanied by the disappearance of the P-wave. Particularly, the diagnosis of paroxysmal atrial fibrillation requires 24 hours of dynamic electrocardiogram monitoring and diagnosis, and electrode plates are required to be attached to the skin of a tested person, so that the problems of complex operation, less application scene and high detection cost exist.
The widely existing PPG (Photoplethysmography) technology on a moving bracelet is to record the tiny change of skin color caused by the blood volume change of arteries and capillaries under the skin through a bracelet photoelectric receiving tube, then extract pulse wave through a specific algorithm, and the blood flow of a part to be measured is unobstructed and is tightly attached to a photoelectric receiver without light leakage. The vPPG (video Photoplethysmography ) is a video-based photoelectric volume description method, and the micro change of skin color caused by the blood volume change of a facial capillary is recorded by a mobile phone camera or a common consumer-level camera, so that the method has the advantages of low cost, non-contact, simplicity and convenience, and can be used for monitoring the heart rate of the aged or the infirm infant during sleeping.
The traditional atrial fibrillation detection algorithm needs to extract the characteristics of the time domain, the frequency domain or the time-frequency domain of the interval (peak interval) of the ECG signal RR, such as fast Fourier transform, wavelet transform and the like, then screen out more important characteristics, and classify the important characteristics by classification methods based on threshold values, nearest neighbor, support vector machine, decision tree and the like. Therefore, the traditional atrial fibrillation detection has the problems of complicated feature extraction steps, high detection cost and low efficiency. In addition, the existing deep learning atrial fibrillation detection method is generally based on a convolutional neural network to learn a large number of ECG data sets, and aims at vPPG data, a large public atrial fibrillation patient data set is lacked, and the learning efficiency is low. No means or technique for solving or improving the above problems is currently known.
Disclosure of Invention
Aiming at the defects or improvement demands of the existing method, the invention provides an end-to-end non-contact atrial fibrillation automatic detection system and method based on vPGG signals, which solve the problems that in the prior art, the occurrence of atrial fibrillation is not considered in the process of remotely and non-contact extracting pulse waves from face videos, the time and space correlation of the pulse signals are not considered at the same time, the detection precision is low due to the fact that irregular fragments are not considered to occupy larger weight in model training, and the detection cost is high, the operation is complex and the method is not suitable for daily monitoring by using 12-lead ECG.
To achieve the above object, the present invention provides an end-to-end non-contact atrial fibrillation automatic detection system based on vpg signals, comprising: the data acquisition module is used for recording face videos of users to be detected; the data preprocessing module is used for removing the position with large noise interference at the beginning and the end of the recorded video, downsampling, intercepting the recorded video into a uniform length and a uniform size, and manufacturing corresponding one-hot labels aiming at atrial fibrillation data and non-atrial fibrillation data; the facial video pulse wave extraction module is used for extracting pulse wave signals of the processed facial video through the P3D convolutional neural network; the data noise reduction module is used for removing noise from vPPG data based on the neural network of the FCN-DN; the atrial fibrillation detection module is used for training a model to enable the model to learn to divide atrial fibrillation fragments and non-atrial fibrillation fragments;
the pulse wave extraction module is used for extracting pulse waves through P3D convolution network training, the P3D convolution network is compared with a 2D convolution layer, the space and time dimension characteristics of adjacent frames of video are extracted at the same time, and the characteristics of a residual error network are used for reference, so that the calculation efficiency is high.
The data noise reduction module comprises an encoder and a decoder, wherein the encoder consists of a cascade convolution layer and a batch standardization layer, compression noise reduction of the vPPG signal is realized, the decoder has the structure of antisymmetry of the encoder, consists of a deconvolution layer and an activation layer, and rebuilds the vPPG signal;
the atrial fibrillation detection module divides a plurality of vPPG signals marked with atrial fibrillation and non-atrial fibrillation into a training set and a testing set, the training set is input into an atrial fibrillation detection model, the testing set verifies accuracy, and parameters are finely adjusted to obtain a trained atrial fibrillation detection model; the atrial fibrillation detection model is a training model combining a long-short-time memory network based on an attention mechanism and a convolution network;
the atrial fibrillation detection module is used for inputting the face video to be detected into the data preprocessing module, the pulse wave extraction module and the data noise reduction module to obtain a vPPG signal segment; and inputting the vPPG signal segment into a trained atrial fibrillation detection model so as to judge whether the vPPG signal to be detected contains the atrial fibrillation segment.
Further, the atrial fibrillation detection module includes: the shape conversion layer, two combined training networks, the fusion layer and the classification layer are connected in sequence; one of the two combined training networks is a long and short term memory network (LSTM) block, and the other is a convolutional network block.
Further, the size of the input layer is 1024×1. The long-short-time memory network block includes: a basic long-short time memory layer, a long-short time memory layer based on an attention mechanism and a downsampling layer.
Further, the convolutional network block includes: one-dimensional convolution layer, batch standardization layer and maximum pooling layer. The one-dimensional convolution layers are highly parallel, can extract the intrinsic characteristics of input data, and are formed by connecting 128 x 8, 256 x 5 and 128 x 3 three one-dimensional convolution layers with a batch standardization level.
Furthermore, the fusion layer is used for splicing the weights generated by the convolution network module and the long-short-time memory network module.
Further, the classification layer is a full-connection layer, and classifies the data into atrial fibrillation fragments and non-atrial fibrillation fragments through a softmax function.
Further, training parameters of the model are updated using an Adam optimizer, bachSize size is 128, and the loss function is a cross entropy loss function.
The invention provides an end-to-end non-contact atrial fibrillation detection system based on a vPPG signal, which is used in the field of medical data analysis, and also provides an end-to-end non-contact atrial fibrillation automatic detection method based on the vPPG signal.
Compared with the prior art, the scheme provided by the invention has the following beneficial effects:
1) The invention provides a remote non-contact atrial fibrillation automatic detection system based on a vPPG signal of a face video, which comprises a face video preprocessing module, a pulse wave extraction module, a data noise reduction module and an atrial fibrillation detection module; the face video preprocessing module is used for downsampling data, intercepting fragments with uniform sizes and manufacturing corresponding labels; the pulse wave extraction module learns facial pulse waves through a P3D convolution network; the automatic noise reduction module removes noise from the collected vPPG signals through the encoder and the decoder; the atrial fibrillation detection module is a parallel network of a long-short-time memory network and a convolution network of a joint attention mechanism. According to the invention, the atrial fibrillation fragments can be accurately identified by training atrial fibrillation patients collected in the Anhui province standing hospital.
2) Compared with the existing atrial fibrillation automatic detection system based on the machine learning and deep learning algorithm, the atrial fibrillation detection model is trained by collecting atrial fibrillation patient data through a hospital, pulse waves are extracted from face videos through a P3D convolution network, noise interference is removed through an FCN-DN noise reduction network, the complicated steps of manually setting a threshold value for filtering noise and manually selecting atrial fibrillation data characteristics for the atrial fibrillation data are omitted, whether the atrial fibrillation occurs or not can be directly detected from the face videos, and the atrial fibrillation detection system has the characteristics of being long-range, non-contact, accurate and efficient.
3) The parallel network combining the long-short-time memory network containing the attention mechanism and the convolution network enables the model to extract the time front-back characteristics of the data single variable through the long-short-time memory network containing the attention mechanism, extract the global characteristics of the data multivariable through the convolution network, and realize complementation of the two networks, and has comprehensive detection, high precision and good effect.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate and do not limit the invention.
Fig. 1 is a schematic structural diagram of an end-to-end non-contact atrial fibrillation automatic detection system based on vpg signals according to an embodiment of the present invention;
fig. 2 is an algorithm flow chart of an end-to-end non-contact atrial fibrillation automatic detection system based on vpg signal according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a model of a P3D convolutional network according to one embodiment of the present invention;
fig. 4 is a schematic diagram of an FCN-DN noise reduction network according to an embodiment of the present invention;
FIG. 5 is a block diagram of a parallel network model of a long and short time memory network and a convolutional network of a joint attention mechanism provided by an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail hereinafter with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be arbitrarily combined with each other.
The invention provides an end-to-end remote noncontact atrial fibrillation detection automatic detection system based on a vPPG signal, which is shown in figure 1 and comprises: the device comprises a face video preprocessing module, a pulse wave extraction module, a data noise reduction module and an atrial fibrillation detection module.
The face video preprocessing module is used for recording face videos of people for 2 minutes. Preferably, a common mobile phone camera or a consumer grade camera is adopted; during recording, the head of the person is kept as free from intense shaking as possible. Secondly, removing places with larger noise interference at the beginning and the end of face videos acquired at the positions (such as indoor places and hospitals) and downsampling, intercepting the signals to a uniform length and a uniform size, manufacturing corresponding labels one-hot labels according to the acquired fingertip pulse waves aiming at atrial fibrillation and non-atrial fibrillation data, reducing the resolution of the videos, and trimming the size of images of each frame in the videos.
And the pulse wave extraction module is used for extracting a vPPG signal from the processed face video recorded by the camera through the P3D convolutional neural network. The extraction method specifically includes that face videos are used as training sets and input into a P3D convolution network, real pulse waves of fingertips are used as labels, a random gradient descent algorithm is used as an optimization algorithm, and mean square error is used as a loss function. Training the network to learn and generate a corresponding human face video pulse wave vPPG.
And the data denoising module is used for removing noise from the vPPG data based on a neural network of the FCN-DN (full-connection noise reduction self-coding). Self-encoders are commonly used in the fields of speech signal noise reduction, data compression, and the like. The module comprises an encoder and a decoder, wherein the encoder consists of a cascade convolution layer and a batch standardization layer, compression noise reduction of the vPPG signal is realized, the structure of the decoder is antisymmetry of the encoder, the decoder consists of a deconvolution layer and an activation layer, and the vPPG signal is rebuilt.
And the atrial fibrillation detection module is used for training a model to enable the model to learn to divide atrial fibrillation fragments and non-atrial fibrillation fragments. Firstly, dividing a plurality of vPPG signals marked with atrial fibrillation and non-atrial fibrillation into a training set and a testing set, inputting the training set into an atrial fibrillation detection model, verifying accuracy by the testing set, and finely adjusting parameters to obtain a trained atrial fibrillation detection model. The atrial fibrillation detection model is a atrial fibrillation detection model combining a long-short-time memory network based on an attention mechanism and a convolution network. And inputting the preprocessed face video to be detected into a pulse wave extraction module and a data noise reduction module to obtain a vPPG signal segment. And inputting the vPPG signal segment into a trained atrial fibrillation detection model so as to judge whether the vPPG signal to be detected contains the atrial fibrillation segment.
Fig. 2 shows an algorithm flow chart of an end-to-end non-contact atrial fibrillation automatic detection system based on vpg signals. As shown in fig. 2, the end-to-end non-contact atrial fibrillation automatic detection method based on the vpg signal includes the following steps:
s1, collecting face videos. And recording the face video of the user to be detected for 2 minutes by using a face video acquisition module.
S2, preprocessing the recorded face video by using a face video preprocessing module.
S3, judging whether the atrial fibrillation detection model is trained, if not, turning to step S41, and if so, turning to step S51.
S41, dividing a plurality of pieces of data marked with atrial fibrillation and non-atrial fibrillation into a training set and a testing set;
s42, the pulse wave extraction module and the data noise reduction module are built, the pulse wave extraction module is used for extracting vPPG signals from training data, and the data noise reduction module is used for removing noise from the vPPG data.
S43, inputting the vPPG signals which are processed in the training set and the testing set correspondingly into a atrial fibrillation detection module for atrial fibrillation detection, obtaining a trained atrial fibrillation detection model, storing model parameters, and returning to the step S1.
S51, reading a trained atrial fibrillation detection model and corresponding model parameters thereof;
s52, after the facial video data obtained in the S2 are processed by the pulse wave extraction module and the data noise reduction module, the processed vPPG signal is input into a loaded atrial fibrillation detection model, a detection result is obtained through automatic detection of the model, the result is output, and the step S1 is returned.
Further, the pulse wave extraction module extracts pulse waves from the face video clips based on the P3D convolution network. The P3D convolutional network may be placed at the FCN pre-stage or post-stage of noise reduction as needed. Wherein the convolution kernel sizes of the network blocks in the P3D convolution network are 1 x 1 (I), 1 x 3 (S) and 3 x 1 (T) respectively, the structure of the network block is shown in fig. 3, and the residual unit of the residual network is replaced by the P3D network block by referring to the residual network. The P3D network is firstly convolved in the image space and then convolved in the time dimension, as shown in a formula (1)
(I+T·S)·X t ：＝X t +T(S(X t ))＝X t+1 (1)
Wherein S (·) represents the convolution of the image space, T (·) represents the convolution in the time dimension, I is the identity matrix, X t The output at time t is shown.
Further, as shown in fig. 4, the data denoising module denoises the vpg data segment based on the encoder and decoder network. Wherein the encoders are made of different rollsThe convolution block is formed by cascade connection, and consists of convolution layers and batch normalization layers. Wherein the batch normalized activation function is a reLu function. Encoders are commonly used in the field of data compression and speech signal denoising, which compress and map input data x to a potential spatial representation z, implementing data compression and noise reduction. The decoder is antisymmetric of the encoder, and consists of different deconvolution blocks, which consist of deconvolution layers, batch normalization layers. The decoder reconstructs the spatial representation z as an output by
As shown in formulas (2) and (3), wherein f (·) and g (·) represent nonlinear activation functions, sigmoid functions and reLu functions are commonly used. W, b represents the encoder weight matrix and the bias matrix. />
Representing the weight matrix and the bias matrix of the decoder. The target loss function is (4). The module takes a vPPG signal sample added with random noise as input, takes a network without noise as a reference, and learns the characteristics of the noise through training so as to remove the noise.
z＝f(Wx+b) (2)
Fig. 5 is a schematic diagram of an atrial fibrillation detection module with a training network employing a parallel network of a long and short memory network and a convolutional network of a joint attention mechanism. The attention mechanism recognizes information important for atrial fibrillation detection by weighted average of states of hidden layers of the long-short-term memory network.
The atrial fibrillation detection module comprises: the shape transformation layer, two combined training networks (one is a long and short time memory network (LSTM) block, the other is a convolution network block), the fusion layer and the classification layer. The size of the Input layer (Input transform block in fig. 5) is 1024 x 1.
The long and short term memory network block comprises: a basic long short time memory Layer (LSTM), a long short time memory layer (attention+lstm) based on an Attention mechanism, a downsampling layer. The long-short-time memory network is used for extracting the intrinsic characteristics of the time sequence signals and memorizing the characteristics of the long sequences without forgetting. Attention mechanisms are classified into many categories, first in computer vision, and then widely applied to the field of natural language processing; the method can increase the weight aiming at the network parameters for improving the classification effect, and notice the interested places, so that the model learning rate is faster and the effect is better.
The convolutional network block comprises: one-dimensional convolution layer, batch standardization layer and maximum pooling layer. The one-dimensional convolution layers are highly parallel, can extract the intrinsic characteristics of input data, and are formed by 128 x 8, 256 x 5 and 128 x 3 three one-dimensional convolution layers and batch standardization layer cascade, and the activation functions are relu functions. And the batch standardization layer standardizes the distribution of the middle layer of the neural network, improves the gradient disappearance problem and improves the learning rate.
The fusion layer Concate is used for splicing the weights generated by the convolution network module and the long-short-time memory network module. The classification layer is a fully connected layer, classifies the data into atrial fibrillation fragments and non-atrial fibrillation fragments by a softmax function.
The optimizer in the atrial fibrillation detection process uses Adam optimizer to update training parameters, bachSize size is 128, and the loss function is cross entropy loss function.
The long-short-time memory network of the attention mechanism consists of a basic LSTM network layer, and an attention mechanism-based LSTM layer and a Dropout layer, wherein the Dropout layer prevents the model from being fitted excessively.
LSTM is one of RNN (recurent neural network, cyclic convolutional network) networks, which can solve the problem of RNN network long-distance gradient extinction. LSTM comprises three gates, including a forget gate f t Input gate i t And an output gate o t Forgetting, inputting and outputting information are determined through a sigmoid function. Specific tThe time is calculated as follows:
f t ＝δ(W f ·[h t-1 ，x t ]+b f ) (5)
i t ＝δ(W i ·[h t-1 ，x t ]+b i ) (6)
o t ＝δ(W o ·[h t-1 ，x t ]+b o ) (7)
h t ＝o t *tanh(C t ) (10)
wherein h is t Represent hidden layer, C t Representing layers of cellular status, which carry information. Wherein W is f ，W i ， W o ，W c Is a weight matrix. b f ，b i ，b o ，b c Is a bias matrix, and the tanh function is the normalization of the data to [ -1,1]Between them.
The attention mechanism calculates the context vector c by making the weights related to atrial fibrillation larger, in particular by means of weighted summation i 。
Wherein h is a hidden layer of the LSTM layer, a ij Is the weight of each hidden layer, c i Is a context vector for time T, T m Is the total time;
wherein e ij Is near the calculated position jThe degree of matching, i.e. similarity,
e ij ＝score(h i-1 ，h j ) (13)
score＝h i-1 T h j (14)
finally, the long-short-time memory network and the convolution network of the attention mechanism are connected together through a fusion layer, and finally, the classification layer predicts the occurrence probability of atrial fibrillation through a softmax function by a full-connection layer, and the trained loss function is a cross entropy loss function.
The cross entropy loss function is:
where L represents the loss function, N represents the total number of samples of the vpg signal segment, y (i) The label representing the sample i is displayed,
representing the model's prediction of the probability of occurrence of sample i.
As an embodiment, a data set is manufactured by utilizing a atrial fibrillation patient vPPG signal and a normal person vPPG signal which are collected in cooperation with a hospital, and a weight matrix and a bias term of an atrial fibrillation detection model are iteratively trained as input, so that a trained atrial fibrillation detection model is finally obtained.
Through cooperation with an Anhui province standing hospital, a database of vPPG signals of atrial fibrillation patients is established, and face videos and corresponding fingertip pulse waves of 80 atrial fibrillation patients are acquired and used for detecting atrial fibrillation. Through verification, the automatic detection result is accurate, and the detection process is efficient and convenient.
The automatic detection method of the terminal-to-terminal atrial fibrillation based on the vPPG signal aiming at the facial video, provided by the invention, ensures that the neural network pays attention to the learning of the atrial fibrillation irregular fragments, and the learning rate is faster; the internal features of the time sequence can be learned based on a long and short-time memory network, and the global features of the sequence can be learned in parallel based on a convolution network, so that the atrial fibrillation detection is applicable to more scenes, and has higher accuracy and lower cost.
Although the embodiments of the present invention are described above, the embodiments are only used for facilitating understanding of the present invention, and are not intended to limit the present invention. Any person skilled in the art to which the present invention pertains may make any modifications, changes, equivalents, etc. in form and detail of the implementation without departing from the spirit and principles of the present invention disclosed herein, which are within the scope of the present invention. Accordingly, the scope of the invention should be determined from the following claims.
Claims (8)
1. An end-to-end non-contact atrial fibrillation automatic detection system based on a vpg signal, comprising: the data preprocessing module is used for recording face videos of users to be detected, removing the positions with large noise interference at the beginning and the end of the recorded videos, and intercepting the recorded videos into uniform lengths; the pulse wave extraction module is used for extracting a vPPG signal of the face video through the P3D convolutional neural network; the data noise reduction module is used for removing noise from vPPG data based on the neural network of the FCN-DN; the atrial fibrillation detection module is used for training a model to enable the model to learn to divide atrial fibrillation fragments and non-atrial fibrillation fragments;
the pulse wave extraction module is a residual network formed by a P3D convolution neural network, the P3D convolution network convolves in an image space firstly, and convolves in a time dimension, and the convolution kernel sizes of network blocks in the P3D convolution network are 1 x 1,1 x 3 and 3 x 1 respectively;
the data noise reduction module comprises an encoder and a decoder, wherein the encoder consists of a cascade convolution layer and a batch standardization layer, compression noise reduction of the vPPG signal is realized, the structure of the decoder is antisymmetry of the encoder, the decoder consists of a deconvolution layer and an activation layer, and the vPPG signal is reconstructed;
the atrial fibrillation detection module divides a plurality of vPPG signals marked with atrial fibrillation and non-atrial fibrillation into a training set and a testing set, the training set is input into an atrial fibrillation detection model, the testing set verifies accuracy, and parameters are finely adjusted to obtain a trained atrial fibrillation detection model;
the atrial fibrillation detection module is used for inputting the preprocessed face video to be detected into the pulse wave extraction module and the data noise reduction module to obtain a vPPG signal segment; and inputting the vPPG signal segment into a trained atrial fibrillation detection model so as to judge whether the vPPG signal to be detected contains the atrial fibrillation segment.
2. The system of claim 1, wherein the atrial fibrillation detection module comprises: an input layer, two combined training networks, a fusion layer and a classification layer; one of the two combined training networks is a long and short memory network LSTM block, and the other is a convolution network block.
3. The system of claim 2, wherein the long-short-time memory network block comprises: a basic long-short time memory layer, a long-short time memory layer based on an attention mechanism and a downsampling layer.
4. A system according to claim 3, wherein said convolutional network block comprises: one-dimensional convolution layer, batch standardization layer and maximum pooling layer.
5. The system of claim 4 wherein the one-dimensional convolution layers are highly parallel and extract intrinsic features of the input data, and are comprised of 128 x 8, 256 x 5, 128 x 3 three one-dimensional convolution layers in series with a normalization hierarchy.
6. The system of claim 5, wherein the fusion layer is a concatenation of weights generated by the convolutional network module and the long-short term memory network module.
7. The system of claim 5, wherein the classification layer is a fully connected layer, and the data is classified into atrial fibrillation fragments and non-atrial fibrillation fragments by a softmax function.
8. The system of claim 7, wherein training parameters of the model are updated using an Adam optimizer, bachSize size is 128, and the loss function is a cross entropy loss function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011423595.3A CN112587153B (en) | 2020-12-08 | 2020-12-08 | End-to-end non-contact atrial fibrillation automatic detection system and method based on vPPG signal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011423595.3A CN112587153B (en) | 2020-12-08 | 2020-12-08 | End-to-end non-contact atrial fibrillation automatic detection system and method based on vPPG signal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112587153A CN112587153A (en) | 2021-04-02 |
| CN112587153B true CN112587153B (en) | 2023-06-02 |
Family
ID=75188864
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011423595.3A Active CN112587153B (en) | 2020-12-08 | 2020-12-08 | End-to-end non-contact atrial fibrillation automatic detection system and method based on vPPG signal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112587153B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113080907B (en) * | 2021-04-14 | 2022-10-25 | 贵州省人民医院 | Pulse wave signal processing method and device |
| CN113598741B (en) * | 2021-06-30 | 2024-03-22 | 合肥工业大学 | Atrial fibrillation evaluation model training method, atrial fibrillation evaluation method and atrial fibrillation evaluation device |
| CN113598742B (en) * | 2021-06-30 | 2024-03-22 | 合肥工业大学 | Atrial fibrillation classification model training method, atrial fibrillation identification method and atrial fibrillation classification model training system |
| CN114764581B (en) * | 2022-06-16 | 2022-09-09 | 合肥心之声健康科技有限公司 | Atrial fibrillation classification method, device and system based on RR interphase spatial features |
| CN116035548B (en) * | 2023-03-31 | 2023-06-09 | 深圳市九天睿芯科技有限公司 | Method and device for detecting heart rhythm state, intelligent wearable device and storage medium |
| CN117257324B (en) * | 2023-11-22 | 2024-01-30 | 齐鲁工业大学(山东省科学院) | Atrial fibrillation detection method based on convolutional neural network and ECG (electro-magnetic resonance) signals |
| CN117694845B (en) * | 2024-02-06 | 2024-04-26 | 北京科技大学 | Non-contact physiological signal detection method and device based on fusion characteristic enhancement |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104382575A (en) * | 2014-11-20 | 2015-03-04 | 惠州Tcl移动通信有限公司 | Heart rhythm detection method based on mobile terminal and mobile terminal |
| CN108606798A (en) * | 2018-05-10 | 2018-10-02 | 东北大学 | Contactless atrial fibrillation intelligent checking system based on depth convolution residual error network |
| CN109871808A (en) * | 2019-02-21 | 2019-06-11 | 天津惊帆科技有限公司 | Atrial fibrillation model training and detecting method and device |
| CN110680310A (en) * | 2019-10-21 | 2020-01-14 | 北京航空航天大学 | Electrocardiosignal atrial fibrillation detection method based on one-dimensional dense connection convolution network |
| CN111345803A (en) * | 2020-03-20 | 2020-06-30 | 浙江大学城市学院 | Heart rate variability measuring method based on mobile device camera |
| CN111728602A (en) * | 2020-08-21 | 2020-10-02 | 之江实验室 | Non-contact blood pressure measuring device based on PPG |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10004410B2 (en) * | 2013-12-19 | 2018-06-26 | The Board Of Trustees Of The University Of Illinois | System and methods for measuring physiological parameters |
| JP6531834B2 (en) * | 2015-11-20 | 2019-06-19 | 富士通株式会社 | INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND INFORMATION PROCESSING PROGRAM |
-
2020
- 2020-12-08 CN CN202011423595.3A patent/CN112587153B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104382575A (en) * | 2014-11-20 | 2015-03-04 | 惠州Tcl移动通信有限公司 | Heart rhythm detection method based on mobile terminal and mobile terminal |
| CN108606798A (en) * | 2018-05-10 | 2018-10-02 | 东北大学 | Contactless atrial fibrillation intelligent checking system based on depth convolution residual error network |
| CN109871808A (en) * | 2019-02-21 | 2019-06-11 | 天津惊帆科技有限公司 | Atrial fibrillation model training and detecting method and device |
| CN110680310A (en) * | 2019-10-21 | 2020-01-14 | 北京航空航天大学 | Electrocardiosignal atrial fibrillation detection method based on one-dimensional dense connection convolution network |
| CN111345803A (en) * | 2020-03-20 | 2020-06-30 | 浙江大学城市学院 | Heart rate variability measuring method based on mobile device camera |
| CN111728602A (en) * | 2020-08-21 | 2020-10-02 | 之江实验室 | Non-contact blood pressure measuring device based on PPG |
Non-Patent Citations (2)
| Title |
|---|
| 现实场景中基于手机视频的脉搏波检测;刘雪南;《CNKI优秀硕士学位论文全文库》;20200115(第1期);全文 * |
| 视频脉搏特征的非接触房颤检测;张姁 等;《计算机工程与应用》;20220324;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112587153A (en) | 2021-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112587153B (en) | End-to-end non-contact atrial fibrillation automatic detection system and method based on vPPG signal | |
| CN108304917A (en) | A kind of P300 signal detecting methods based on LSTM networks | |
| CN112508110A (en) | Deep learning-based electrocardiosignal graph classification method | |
| CN111297349A (en) | Machine learning-based heart rhythm classification system | |
| CN110403600B (en) | Intelligent analysis method and system for paroxysmal atrial fibrillation based on difference time scatter diagram | |
| CN110638430B (en) | Method for building cascade neural network ECG signal arrhythmia classification model | |
| CN104636580A (en) | Health monitoring mobile phone based on human face | |
| CN113349793A (en) | Arrhythmia classification method based on layered attention model | |
| CN111248859A (en) | Automatic sleep apnea detection method based on convolutional neural network | |
| CN114067435A (en) | Sleep behavior detection method and system based on pseudo-3D convolutional network and attention mechanism | |
| CN114093501A (en) | Intelligent auxiliary analysis method for children's motor epilepsy based on synchronous video and electroencephalogram | |
| CN112674782B (en) | Device and method for detecting epileptic-like electrical activity of epileptic during inter-seizure period | |
| CN114732409A (en) | Emotion recognition method based on electroencephalogram signals | |
| CN113995419A (en) | Atrial fibrillation risk prediction system based on heartbeat rhythm signal and application thereof | |
| CN112932501A (en) | Method for automatically identifying insomnia based on one-dimensional convolutional neural network | |
| CN115281688A (en) | Cardiac hypertrophy multi-label detection system based on multi-mode deep learning | |
| CN115640507A (en) | Abnormal data screening method based on electrocardio-heart sound joint analysis | |
| CN116524380A (en) | Target detection method based on brain-computer signal fusion | |
| CN114521900B (en) | Arrhythmia classification and identification method based on transfer learning | |
| CN114098714B (en) | Method for establishing frozen gait recognition model based on machine vision | |
| CN115581435A (en) | Sleep monitoring method and device based on multiple sensors | |
| CN115017960A (en) | Electroencephalogram signal classification method based on space-time combined MLP network and application | |
| CN112101095B (en) | Suicide and violence tendency emotion recognition method based on language and limb characteristics | |
| CN107731306A (en) | A kind of contactless heart rate extracting method based on thermal imaging | |
| CN114847968A (en) | Electroencephalogram sleep staging method based on long-term and short-term memory network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |