WO2024108483A1 - Multimodal neural biological signal processing method and apparatus, and server and storage medium - Google Patents

Multimodal neural biological signal processing method and apparatus, and server and storage medium Download PDF

Info

Publication number
WO2024108483A1
WO2024108483A1 PCT/CN2022/134058 CN2022134058W WO2024108483A1 WO 2024108483 A1 WO2024108483 A1 WO 2024108483A1 CN 2022134058 W CN2022134058 W CN 2022134058W WO 2024108483 A1 WO2024108483 A1 WO 2024108483A1
Authority
WO
WIPO (PCT)
Prior art keywords
layer
model
neural
biological signal
deep learning
Prior art date
Application number
PCT/CN2022/134058
Other languages
French (fr)
Chinese (zh)
Inventor
谢津
詹阳
马征
Original Assignee
中国科学院深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院 filed Critical 中国科学院深圳先进技术研究院
Priority to PCT/CN2022/134058 priority Critical patent/WO2024108483A1/en
Publication of WO2024108483A1 publication Critical patent/WO2024108483A1/en

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/372—Analysis of electroencephalograms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Definitions

  • the present application relates to the field of data processing technology, and specifically to a method, device, server and storage medium for processing multimodal neural biological signals.
  • classical machine learning methods are usually used to analyze the complex and variable relationships in neurobiological data such as rsEEG and magnetic resonance imaging (MRI) to identify neurobiological representations after drug treatment.
  • MRI magnetic resonance imaging
  • the classical machine learning computational model can reflect neurobiological characteristics to a certain extent, it relies on artificial experience to design or select features. It is difficult to effectively analyze the complex nonlinear relationships in neurobiological signals to determine the neurobiological characteristics of the therapeutic drug response phenotype.
  • the embodiments of the present application provide a method, device, server and storage medium for processing multimodal neurobiological signals to solve the problem that neurobiological characteristics are difficult to accurately predict.
  • the method for processing multimodal neural biological signals is used in a deep learning network, wherein the deep learning network includes a pre-built deep learning model and a deep regression model, wherein the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer, and the method includes: obtaining a multimodal neural biological signal to be processed, preprocessing the multimodal neural biological signal to obtain a preprocessed multimodal neural biological signal; inputting the preprocessed multimodal neural biological signal into the deep learning model, and extracting deep features of each modality of neural biological signal in the multimodal neural biological signal based on the deep learning model; inputting a plurality of the deep features into the feature fusion layer for feature fusion to obtain a target fusion feature; inputting the target fusion feature into the regression layer after passing through the fully connected layer, and using the regression layer to perform sign prediction on the target fusion feature to generate a biological sign prediction result.
  • the method for processing multimodal neural biological signals uses a deep learning network to extract features from multimodal neural biological signals and fuse multiple deep features, and then predicts signs of the target fused features obtained by fusion to obtain biological feature prediction results.
  • multiple deep features can be fused based on objective multimodal neural biological signals to achieve effective capture of multiple deep features without artificial design or feature selection
  • the deep learning network has a multi-level nonlinear structure, which can effectively analyze the nonlinear relationship in neural biological signals, so that effective biological signs can be analyzed from multiple deep features, achieving effective prediction of biological representations of neural signals and improving the accuracy of biological representation prediction.
  • a plurality of deep learning sub-models are constructed in the deep learning model; the multimodal neural biological signal is input into the deep learning model, and the deep features of each modality of the neural biological signal in the multimodal neural biological signal are extracted based on the deep learning model, including: determining the type of the multimodal neural biological signal; determining the deep learning sub-models corresponding to each type of neural biological signal; and inputting each type of neural biological signal into the corresponding deep learning sub-model respectively to obtain the deep features corresponding to each type of neural biological signal.
  • the multimodal neural biological signal processing method provided in the embodiment of the present application facilitates the directional extraction of deep features of each type of neural biological signal by inputting different types of neural biological signals into corresponding deep learning sub-models for deep feature extraction, thereby improving the extraction accuracy of deep features.
  • the first deep learning sub-model corresponding to the first type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a second sub-model; the first type of neural biological signal is input into the first deep learning sub-model to obtain a deep feature corresponding to the first type of neural biological signal, including: inputting the first type of neural biological signal into the second sub-model for feature extraction to obtain first feature data; fusing the first feature data with a preconfigured position code to obtain fused data; inputting the fused data into the model group for feature extraction to obtain a deep feature corresponding to the first type of neural biological signal.
  • the first type of neural biological signal is an electroencephalogram signal;
  • the first sub-model includes a Transformer model;
  • the second sub-model includes a convolutional neural network model;
  • the convolutional neural network includes a convolutional layer and an average pooling layer;
  • the Transformer model includes an attention module and a feedforward network module;
  • the attention module includes a multi-head attention layer and a first normalization layer;
  • the feedforward network module includes a fully connected feedforward layer and a second normalization layer;
  • the multi-head attention layer is connected to the first normalization layer;
  • the fully connected feedforward layer is connected to the second normalization layer;
  • the first normalization layer is connected to the fully connected feedforward layer;
  • the attention module and the feedforward network module have corresponding shortcut connections.
  • the method for processing multimodal neural biological signals constructs a first deep learning sub-model for EEG signals, and the first deep learning sub-model is composed of a multi-layer deep structure (a model group consisting of a first sub-model and multiple second sub-models), thereby being able to perform high-dimensional feature extraction on EEG signals, thereby being able to more accurately mine and extract the spatiotemporal characteristics and long-range dependencies of EEG signals.
  • a multi-layer deep structure a model group consisting of a first sub-model and multiple second sub-models
  • the second deep learning sub-model corresponding to the second type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a third sub-model, and the third sub-model includes a convolution layer, a maximum pooling layer and a fully connected layer;
  • the second type of neural biological signal is input into the second deep learning sub-model to obtain a deep feature corresponding to the second type of neural biological signal, including: inputting the second type of neural biological signal into the convolution layer of the third sub-model for multi-dimensional convolution processing to obtain a convolution processing result; inputting the convolution processing result into the model group for feature extraction, and passing through the maximum pooling layer and the fully connected layer, outputting the deep feature corresponding to the second type of neural biological signal.
  • the second type of neural biological signal is a functional magnetic resonance signal;
  • the first sub-model includes a Transformer model;
  • the third sub-model includes a point 4D convolutional network model;
  • the point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer and a fully connected layer;
  • a model group composed of multiple Transformer models connected in series is arranged between the point 4D convolutional layer and the maximum pooling layer;
  • the Transformer model includes an attention module and a feedforward network module;
  • the attention module includes a multi-head attention layer and a first normalization layer;
  • the feedforward network module includes a fully connected feedforward layer and a second normalization layer;
  • the multi-head attention layer is connected to the first normalization layer;
  • the fully connected feedforward layer is connected to the second normalization layer;
  • the first normalization layer is connected to the fully connected feedforward layer;
  • the attention module and the feedforward network module have corresponding shortcut connections.
  • the method for processing multimodal neurobiological signals constructs a second deep learning sub-model for functional magnetic resonance signals, and the second deep learning sub-model is composed of a multi-layer deep structure (a model group consisting of a third sub-model and multiple second sub-models), thereby realizing high-dimensional feature extraction of functional magnetic resonance signals, thereby being able to more accurately mine and extract the spatial features and functional connectivity features of functional magnetic resonance signals.
  • the target fusion feature is input into the regression layer after passing through the fully connected layer, and the target fusion feature is predicted by the regression layer, before generating a biological sign prediction result, it also includes: obtaining the data type corresponding to the target fusion feature; constructing a loss function corresponding to the data type; using the loss function to optimize the parameters of the deep regression network, and the regression layer is deployed in the deep regression network.
  • the multimodal neural biological signal processing method facilitates the joint learning optimization of the target fusion feature learning loss and the regression loss by constructing a loss function, so as to improve the regression performance of the regression layer and ensure the analysis accuracy of the regression layer.
  • represents the regularization term; ⁇ , ⁇ , ⁇ , ⁇ represent network parameters.
  • the method for processing multimodal neural biological signals provided in the embodiment of the present application adds a regularization term when constructing the loss function to correct the loss function, thereby minimizing overfitting.
  • the method also includes: obtaining the numerical range of the biological sign prediction result; quantizing the numerical range, dividing the numerical range into a number of intervals, and obtaining prediction levels corresponding to the several intervals.
  • the multimodal neural biological signal processing method provided in the embodiment of the present application can assist clinicians in effectively evaluating biological representations by quantifying the numerical range to divide it into several intervals and setting different prediction levels for different intervals.
  • the method further includes: extracting biological features from the biological sign prediction results based on a preset explainable artificial intelligence method; performing replication analysis and convergence analysis on the biological features, and determining individual biomarkers from the biological features.
  • the multimodal neurobiological signal processing method provided in the embodiment of the present application applies explainable artificial intelligence to the biological sign prediction results to extract biological characteristics, and determines individualized biomarkers through replication analysis and convergence analysis. It has great neurobiological significance for the portability and stability of biological characteristics, and can provide potential powerful assistance to clinicians facing mental illness.
  • an embodiment of the present application provides a multimodal neural biological signal processing device for use in a deep learning network, wherein the deep learning network includes a pre-built deep learning model and a deep regression model, wherein the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer, and the device includes: an acquisition module, used to acquire the multimodal neural biological signal to be processed, pre-process the multimodal neural biological signal, and obtain the pre-processed multimodal neural biological signal; a feature extraction module, used to input the pre-processed multimodal neural biological signal into the deep learning model, and extract the deep features of each modality of the multimodal neural biological signal in the multimodal neural biological signal based on the deep learning model; a feature fusion module, used to input a plurality of the deep features into the feature fusion layer for feature fusion, and obtain a target fusion feature; a prediction module, used to input the target fusion feature into the regression layer after passing through the fully connected layer, and use the regression layer to perform sign prediction on the
  • an embodiment of the present application provides a server, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the method for processing multimodal neural biological signals described in the first aspect or any embodiment of the first aspect.
  • an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method for processing multimodal neural biological signals described in the first aspect or any embodiment of the first aspect.
  • FIG1 is a flow chart of a method for processing multimodal neural biosignals according to an embodiment of the present application
  • FIG2 is a schematic diagram of a deep learning network according to an embodiment of the present application.
  • FIG3 is another flow chart of a method for processing multimodal neural biosignals according to an embodiment of the present application.
  • FIG4 is another schematic diagram of a deep learning network according to an embodiment of the present application.
  • FIG5 is a schematic diagram of the fusion of a CNN-Trans model according to an embodiment of the present application.
  • FIG6 is a schematic diagram of the fusion of a 4D-Trans model according to an embodiment of the present application.
  • FIG7 is another flow chart of a method for processing multimodal neural biosignals according to an embodiment of the present application.
  • FIG8 is a schematic diagram of determining a biometric feature according to an embodiment of the present application.
  • FIG9 is a schematic diagram of a clinical auxiliary application according to an embodiment of the present application.
  • FIG10 is a schematic diagram of a deep learning system according to an embodiment of the present application.
  • FIG11 is a structural block diagram of a multimodal neural bio-signal processing device according to an embodiment of the present application.
  • FIG. 12 is a schematic diagram of the hardware structure of a server provided in an embodiment of the present application.
  • an embodiment of a method for processing a multimodal neurobiological signal is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
  • a method for processing multimodal neural biological signals is provided, which can be used in a deep learning network.
  • the deep learning network includes a pre-built deep learning model and a deep regression model.
  • the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer.
  • FIG. 1 is a flow chart of a method for processing a multimodal neural biosignal according to an embodiment of the present application. As shown in FIG. 1 , the process includes the following steps:
  • Multimodal neurobiological signals are various types of neurobiological signals collected by acquisition equipment from people with mental illness.
  • Multimodal neurobiological signals can be obtained from clinical databases, which contain neurobiological signal datasets constructed by multiple research sites for patients with mental illness.
  • the multimodal neurobiological signals may include electroencephalogram (EEG) signals, functional magnetic resonance imaging (fMRI) signals, and behavioral scale data.
  • EEG electroencephalogram
  • fMRI functional magnetic resonance imaging
  • behavioral scale data behavioral scale data
  • the amplitude threshold method is used to remove the artifact segment, that is, the artifact segment is removed by detecting whether the EEG amplitude exceeds the set threshold.
  • Threshold processing is performed on the spatial correlation between channels to remove bad channels, and the EEG of the bad channel is interpolated from the EEG of the adjacent channels using the interpolation method. Since the EEG of the normal channel has a certain correlation with the EEG of the adjacent channel, but the EEG of the bad channel is not correlated with the EEG of the adjacent channel or the correlation is extremely low, the bad channel can be removed by calculating the correlation of the EEG between adjacent channels. For example, the Pearson correlation coefficient between adjacent channels is calculated. When the Pearson correlation coefficient is lower than the threshold, it indicates that there is a bad channel and it is removed.
  • the EEG of the bad channel may be unavailable due to interference and noise introduced during the acquisition process, if the EEG of the bad channel is directly discarded, the number of EEG leads will be reduced and insufficient.
  • the EEG of the bad channel can be simulated by using the interpolation method (such as spherical spline interpolation, cubic spline interpolation, etc.).
  • ICA Independent Component Correlation Algorithm
  • preprocessing can be performed based on the FSL tool.
  • the specific steps include:
  • the fMRI corresponding functional images are realigned to the structural images, and the MRI T1 images are nonlinearly normalized and connected to the standard template of the MRI T1 images of the MNI standard brain.
  • the FSL tool can be used to directly segment brain tissue types such as white matter, gray matter, and cerebrospinal fluid, and then the interference signals corresponding to white matter and cerebrospinal fluid are regressed from the motion-corrected functional image through a regression algorithm (for example, the automatic regressive moving average model ARMA).
  • a regression algorithm for example, the automatic regressive moving average model ARMA.
  • the cutoff point can be set according to the actual usage scenario, and the default value can also be selected.
  • the behavior scale data includes: age, heart rate, blood pressure, gender, race and scale assessment data, etc.
  • Preprocessing the behavior scale data means digitizing and quantifying the behavior scale data to generate a corresponding feature matrix.
  • the above-mentioned multimodal neurobiological signals are not limited to EEG, fMRI and behavioral scale data, but may also include magnetoencephalography (MEG), near-infrared spectroscopy (NIRS), diffusion tensor imaging (DTI), positron emission tomography (PET), eye movement, blood samples, genes and other data. No specific limitation is made here, and technical personnel in this field can determine according to actual needs.
  • MEG magnetoencephalography
  • NIRS near-infrared spectroscopy
  • DTI diffusion tensor imaging
  • PET positron emission tomography
  • multimodal neurobiological signals include control data of psychiatric drugs and placebos.
  • the deep learning model is a model obtained by pre-training multimodal neural biological signals.
  • the deep learning model takes multimodal neural biological signals as input and outputs deep features.
  • the deep learning model consists of a multi-layer deep structure.
  • the pre-processed multimodal neural biological signals are input into the deep learning model, and the deep features of the neural biological signals for each modality are extracted through the deep learning model.
  • the multiple deep features extracted from the multimodal neural biological signals are input into the feature fusion layer in the deep regression model, and the multiple deep features are fused through the feature fusion layer to obtain the fused target fusion features.
  • the biological sign prediction result is used to characterize the prediction results before and after the drug treatment of mental illness.
  • the biological sign prediction result is characterized by a prediction label label, and the prediction label label is calculated based on the difference in the sign scale values before and after the drug treatment of mental illness.
  • the feature fusion layer is connected to the regression layer through two fully connected layers.
  • the target fusion features output by the feature fusion layer are input into the regression layer of the deep regression model after passing through two fully connected layers to obtain the biological sign prediction results, as shown in Figure 2.
  • the deep regression model is a supervised learning model. In the regression layer, the target fusion features are used to predict biological signs and the biological sign prediction results are output.
  • the HAMD 17 data value H0 of each subject's physical sign scale before taking psychiatric drugs is collected in advance. After 8 weeks of medication, the HAMD 17 data H8 of each subject will be collected again.
  • the HAMD 17 value before medication minus the HAMD 17 value after 8 weeks of medication is the biological sign prediction result.
  • the deep features of multimodal neural biological signals can be associated with the efficacy of psychiatric drugs, achieving accurate prediction of the efficacy of psychiatric drugs, and outputting the corresponding biological sign prediction results so that the prediction results can be evaluated.
  • the method for processing multimodal neural biological signals uses a deep learning network to extract features from multimodal neural biological signals and fuse multiple deep features, and then predicts the physical signs of the target fused features obtained by fusion to obtain biological feature prediction results.
  • multiple deep features can be fused based on objective multimodal neural biological signals to achieve effective capture of multiple deep features without artificial design or feature selection.
  • the deep learning network has a multi-level nonlinear structure and can effectively analyze the nonlinear relationship in neural biological signals, so that effective biological signs can be analyzed from multiple deep features, achieving effective prediction of biological representations of neural signals and improving the accuracy of biological representation prediction.
  • a method for processing multimodal neural biological signals is provided, which can be used in a deep learning network.
  • the deep learning network includes a pre-built deep learning model and a deep regression model.
  • the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer.
  • FIG3 is a flow chart of a method for processing a multimodal neural biosignal according to an embodiment of the present application. As shown in FIG3 , the process includes the following steps:
  • step S22 may include:
  • multimodal neural biological signals include neural biological signals of multiple modes, and neural biological signals of different modes correspond to different feature information.
  • different types of neural biological signals are distinguished by the feature information of the multimodal neural biological signals.
  • the deep learning sub-model corresponds to the type of neural biological signal, and the deep learning sub-model corresponding to each type of neural biological signal can be determined based on the corresponding relationship.
  • the multimodal neural biological signal includes A signal, B signal and C signal, and A signal, B signal and C signal belong to different types of signals, and the deep learning model includes sub-model a, sub-model b and sub-model c.
  • a signal corresponds to sub-model a; B signal corresponds to sub-model b; C signal corresponds to sub-model c.
  • Deep features are high-dimensional features generated for neural biological signals, and are represented by high-dimensional feature matrices.
  • each type of neural biological signal can be input into the corresponding deep learning sub-model. For example, input signal A into sub-model a, input signal B into sub-model b, and input signal C into sub-model c, thereby obtaining the corresponding deep features.
  • the first deep learning sub-model corresponding to the first type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a second sub-model, thereby realizing a multi-layer deep structure design for the first deep learning sub-model.
  • the first type of neural biological signal is input into the first deep learning sub-model to obtain the deep features corresponding to the first type of neural biological signal, including:
  • the fused data is input into the model group for feature extraction to obtain the deep features corresponding to the first type of neural biological signal.
  • the second sub-model is used to preliminarily extract the neurobiological features of the first type of neurobiological signal.
  • the model group constructed by multiple first sub-models is used to further extract the deep features of the neurobiological signal.
  • the neurobiological features initially extracted by the second sub-model are used as the first feature data and fused with the position code to obtain fused data of the neurobiological features and the position code.
  • the position code in the neurobiological features it is convenient to determine the position of each signal segment in the first type of neurobiological signal or the collected position.
  • the position of each collection electrode can be encoded and processed according to certain coding rules, and fused into the initially extracted neurobiological features, so as to facilitate the subsequent determination of the depth features at the positions of each collection electrode.
  • the fused data obtained above is input into the model group for high-dimensional feature extraction, so as to extract features that can characterize the spatiotemporal features and long-range dependencies of the EEG signal.
  • the features output by the model group are determined as the deep features of the first type of neural biological signal.
  • first sub-model and the second sub-model there is no limitation on the first sub-model and the second sub-model here, as long as the extraction of deep features can be achieved, technical personnel in this field can determine it according to actual needs.
  • the first type of neural biological signal is an electroencephalogram signal
  • the first sub-model includes a Transformer model
  • the second sub-model includes a convolutional neural network model.
  • the convolutional neural network includes a convolutional layer and an average pooling layer.
  • the Transformer model includes an attention module and a feedforward network module.
  • the attention module includes a multi-head attention layer and a first normalization layer
  • the feedforward network module includes a fully connected feedforward layer and a second normalization layer
  • the multi-head attention layer is connected to the first normalization layer
  • the fully connected feedforward layer is connected to the second normalization layer
  • the first normalization layer is connected to the fully connected feedforward layer
  • the attention module and the feedforward network module have corresponding shortcut connections.
  • the Transformer model based on the multi-head attention mechanism has the advantages of wide applicability, good interpretability, and good at capturing long-range dependencies in sequence signals.
  • the Transformer model can be fused with the Convolutional Neural Networks (CNN) model.
  • CNN Convolutional Neural Networks
  • Figure 5 is a block diagram of the CNN-Trans model obtained by fusing the Transformer model with the CNN model.
  • Figure 5(a) is a structural diagram of the Transformer model.
  • the Transformer model includes an attention module and a feedforward network module.
  • the attention module includes a multi-head attention layer, followed by a first normalization layer; the feedforward network module includes a fully connected feedforward layer, followed by a second normalization layer. Quick connections are used around the attention module and the feedforward network module.
  • the attention module includes a multi-head attention layer, which allows the Transformer model to jointly pay attention to information from different positions, such as using an 8-head attention layer (i.e., 8 attention heads).
  • the number of attention layers is not limited here, and those skilled in the art can determine it according to actual needs.
  • FIG5(b) is a structural diagram of a CNN model, which includes a convolutional layer and an average pooling layer.
  • the CNN model fusion position encoding is input into a model group obtained by connecting multiple Transformer models in series to construct a CNN-Trans model structure.
  • a model group obtained by connecting three Transformer models in series is not limited here, and those skilled in the art can determine it according to actual needs.
  • the deep features of EEG signals can be extracted through the CNN-Trans model.
  • the transfer learning method can be used to use the CNN-Trans model trained with normal human EEG samples as the initialization model to effectively improve the efficiency and accuracy of CNN-Trans model training.
  • the second deep learning sub-model corresponding to the second type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a third sub-model, and the third sub-model includes a convolutional layer, a maximum pooling layer and a fully connected layer, thereby realizing a multi-layer deep structure design for the second deep learning sub-model.
  • the second type of neural biological signal is input into the second deep learning sub-model to obtain the deep features corresponding to the second type of neural biological signal, including:
  • the second type of neural biological signal is input into the convolution layer of the third sub-model for multi-dimensional convolution processing to obtain the convolution processing result.
  • the third sub-model is used to perform multi-dimensional convolution processing on the second type of neural biological signal, and the model group formed by connecting multiple first sub-models in series is used to extract deep features from the second type of neural biological signal.
  • the third sub-model performs multi-dimensional convolution processing on the second type of neurobiological features to convert the second type of neurobiological signals into signals represented by multi-dimensional matrices, and the convolution processing results obtained by the multi-dimensional convolution processing are input into the model group for high-dimensional feature extraction to extract the features that can characterize the spatial and functional connection of the functional magnetic resonance signal.
  • the high-dimensional features output by the model group are output through the maximum pooling layer and the fully connected layer to obtain the deep features of the second type of neurobiological signals.
  • first sub-model and the third sub-model there is no limitation on the first sub-model and the third sub-model here, as long as they can realize the extraction of deep features, those skilled in the art can determine them according to actual needs.
  • the second type of neural biological signal is a functional magnetic resonance signal
  • the first sub-model includes a Transformer model
  • the third sub-model includes a point 4D convolutional network model.
  • the point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer, and a fully connected layer; a model group consisting of multiple Transformer models connected in series is set between the point 4D convolutional layer and the maximum pooling layer;
  • the Transformer model includes an attention module and a feedforward network module;
  • the attention module includes a multi-head attention layer and a first normalization layer;
  • the feedforward network module includes a fully connected feedforward layer and a second normalization layer;
  • the multi-head attention layer is connected to the first normalization layer;
  • the fully connected feedforward layer is connected to the second normalization layer;
  • the first normalization layer is connected to the fully connected feedforward layer;
  • the attention module and the feedforward network module have corresponding shortcut connections.
  • Figure 6 is a block diagram of the 4D-Trans model obtained by fusing the point 4D convolutional network model with the Transformer model.
  • the Transformer model includes an attention module and a feedforward network module.
  • the attention module includes a multi-head attention layer followed by a first normalization layer.
  • the feedforward network module includes a fully connected feedforward layer followed by a second normalization layer. Quick connections are used around the attention module and the feedforward network module, as shown in Figure 5. The corresponding two-dimensional operations are converted to three-dimensional/four-dimensional operations as needed through the Transformer model.
  • the point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer and a fully connected layer.
  • the output result of the point 4D convolutional layer of the point 4D convolutional network model is used as the input of the model group obtained by connecting multiple Transformer models in series, and the output result of the model group is input into the maximum pooling layer and the fully connected layer to construct a 4D-Trans model structure.
  • a model group is obtained by connecting three Transformer models in series. There is no limitation on the number of Transformer models connected in series here, and those skilled in the art can determine it according to actual needs.
  • the deep features of functional magnetic resonance signals can be extracted through this 4D-Trans model.
  • a transfer learning method can be used to use the 4D-Trans model trained with normal human fMRI samples as an initialization model to effectively improve the efficiency and accuracy of 4D-Trans model training.
  • the multimodal neural biological signal processing method provided in this embodiment facilitates the directional extraction of deep features of each type of neural biological signal by inputting different types of neural biological signals into corresponding deep learning sub-models for deep feature extraction, thereby improving the extraction accuracy of deep features.
  • a method for processing multimodal neural biological signals is provided, which can be used in a deep learning network.
  • the deep learning network includes a pre-built deep learning model and a deep regression model.
  • the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer.
  • FIG. 7 is a flow chart of a method for processing a multimodal neural biosignal according to an embodiment of the present application. As shown in FIG. 7 , the process includes the following steps:
  • the data types include discrete data and continuous data.
  • the regression layer can perform classification prediction of discrete values or regression prediction of continuous values.
  • the value feature of the target fusion feature can be analyzed to determine the data type of the current target fusion feature.
  • cross-entropy can be used as the loss function to classify the deep regression model of the regression layer, and a softmax activation function can be added after the last fully connected layer to normalize the output value.
  • the data error parameters can be used to construct the loss function (Loss) to optimize the training of the deep regression model.
  • the data error parameters include the mean absolute error, the root mean square error, and the median absolute error.
  • the constructed loss function is:
  • Loss ⁇ *RMSE+ ⁇ *MAE+ ⁇ *MedAE+ ⁇ *
  • Loss represents the loss function
  • RMSE represents the root mean square error
  • MAE represents the mean absolute error
  • MedAE represents the median absolute error
  • represents the regularization term
  • ⁇ , ⁇ , ⁇ , and ⁇ represent network parameters.
  • RMSE, MAE, and MedAE are three different evaluation indicators, these three evaluation indicators are comprehensively considered here and integrated into the Loss function to improve the regression accuracy.
  • is a regularization term related to the network parameters. By adding the regularization term to the Loss function, the degree of overfitting can be reduced to a certain extent.
  • the error back propagation algorithm can be used to calculate the gradient of the loss function to the network parameters, and the network parameters can be updated with the optimization method to reduce the loss function until the loss function is reduced to the minimum value and the network converges.
  • the network parameters at this time are determined to be the optimal network parameters.
  • all parameters of the deep regression network can be optimized through the loss function until the loss function is reduced to the minimum value and the network converges.
  • the various network parameters determined at this time are determined as the optimal network parameters, and the deep regression network model is deployed according to the optimal network parameters.
  • the biological sign prediction results include discrete value prediction results and continuous value prediction results.
  • the minimum and maximum values of the discrete values constitute the numerical range corresponding to the discrete value prediction results
  • the continuous value prediction results are the numerical ranges constituted by the continuous data values.
  • the biological sign prediction results here are equivalent to classification, and different prediction levels can be set here to correspond to the classification results of discrete values. For example, 0-completely ineffective, 1-slightly effective, 2-effective, 3-very effective.
  • the numerical range can be quantified and divided into multiple intervals for classification training and prediction to determine the prediction level corresponding to different intervals.
  • the numerical range is [-5, 15].
  • the numerical range can be divided into 4 intervals, and the corresponding prediction level is set for each interval. Specifically, the interval [-5, 0] is set to 0-completely invalid, the interval [0, 5] is set to 1-slightly effective, the interval [5, 10] is set to 2-effective, and the interval [10, 15] is set to 3-very effective.
  • the prediction results are comprehensively evaluated using accuracy, precision, F1 value and AUC value.
  • the prediction results can be comprehensively evaluated by calculating indicators such as mean absolute error (MAE), mean absolute percentage error (MAPE), mean square error (MSE), root mean square error (RMSE), logarithm of mean square error (MSLE), median absolute error (MedAE) and coefficient of determination (r2score).
  • MSE mean absolute percentage error
  • MSE mean square error
  • RMSE root mean square error
  • MSLE logarithm of mean square error
  • MedAE median absolute error
  • r2score coefficient of determination
  • indicators such as MAE, MAPE, MSE, RMSE, MSLE, and MedAE are used to evaluate the difference between the true value and the predicted value. The smaller the value, the better.
  • r2score is a statistic used to measure the goodness of fit.
  • the maximum value of r2score is 1. The closer the value of r2score is to 1, the better the deep regression model fits the observed value; conversely, the smaller the value of r2score is, the worse the deep regression model fits the observed value.
  • the interpretable artificial intelligence method (XAI) of the deep learning network is used to analyze the biological characteristics of the drug efficacy of mental illness from the biological sign prediction results.
  • the interpretable artificial intelligence method (XAI) is applied to the deep regression model, and the individual level analysis of the biological sign prediction results output by the deep regression model is performed to find the individual level biological characteristics of the drug efficacy response of mental illness.
  • XAI XAI
  • XAI XAI
  • XAI Japanese artificial intelligence methods
  • occlusion analysis methods occlusion analysis methods
  • gradient visualization methods occlusion analysis methods
  • hierarchical correlation propagation methods etc.
  • the hierarchical relevance propagation method is used to extract biological features.
  • the specific implementation process includes: the processed neurobiological signal is input into the deep regression model to obtain the output result of the regression layer (i.e., the current biological sign prediction result), and the correlation of the output result of the regression layer is back-propagated from the output layer of the deep regression model to the input layer based on the LRP correlation criterion, so that the LRP value of all features (data points) of the input data is obtained.
  • the larger the LRP value the greater the contribution of the feature (data point) in the prediction/decision-making process.
  • an LRP heat map can be generated for each subject, and the biological features at the individual level can be extracted through the LRP heat map.
  • S311 performing replication analysis and convergence analysis on the biometrics to determine individual biomarkers from the biometrics.
  • biosignatures were subjected to replication analysis and convergence analysis, as shown in FIG8 .
  • the portability and stability of the biosignatures were evaluated through the replication analysis results and the convergence analysis results to determine whether the biosignatures for drug efficacy prediction can be used as potential biomarkers.
  • the drug efficacy prediction results and biosignatures can be cross-validated by using the results of other non-drug treatments, further verifying the neurobiological significance and clinical significance of the drug efficacy prediction results and biosignatures.
  • replication analysis is to apply the biological sign prediction results and the biological characteristics of drug efficacy response to the local database, and evaluate the portability and stability of drug efficacy prediction and biological characteristics through replication analysis across data sets in the database to determine whether the biological characteristics can be used as biomarkers.
  • the LRP heat map can be obtained by using the above-mentioned LRP method.
  • the biological characteristics with larger LRP values have greater contributions to biological sign prediction, and biological characteristics with large contributions are used as potential biomarkers.
  • the contribution size can be comprehensively evaluated by using the comprehensive contribution value calculated by a variety of deep learning visualization analysis methods. When the comprehensive contribution value corresponding to a certain biological characteristic is greater than the set threshold, the biological characteristic can be used as a potential biomarker.
  • convergence analysis is mainly based on multimodal neurobiological signals before and after taking psychiatric drugs to predict the efficacy of psychiatric drugs.
  • subjects will not only use drug treatment, but also non-drug treatment, such as physical therapy or psychological therapy such as transcranial magnetic stimulation TMS, so the same subject will have treatment data under different treatment methods.
  • Convergence analysis of non-drug treatment data can further verify the neurobiological significance and clinical significance of biological characteristics.
  • convergence analysis includes: comparing and analyzing the data of poor drug efficacy predicted and poor non-drug treatment effect actually, and finding the common points of these data; comparing and analyzing the data of poor drug efficacy predicted and good non-drug treatment effect actually, and finding the difference of data; comparing and analyzing the data of good drug efficacy predicted and poor non-drug treatment effect actually, and finding the difference of data; comparing and analyzing the data of good drug efficacy predicted and good non-drug treatment effect actually, and finding the common points of these data.
  • the common points and differences determined by the above cross-comparison analysis of data of different treatment methods are more biologically meaningful and clinically meaningful.
  • individual-level biomarkers mined through deep learning networks and interpretable artificial intelligence methods provide potential powerful assistance to clinicians in the treatment of mental illness.
  • clinicians can replicate and analyze the clinical data collected in real time through deep learning networks and biomarkers determined by interpretable artificial intelligence methods.
  • they can further verify, optimize and correct the deep learning model and deep regression model in the deep learning network to achieve cross-validation.
  • the multimodal neurobiological signal processing method by constructing a loss function, facilitates the joint learning optimization of the target fusion feature learning loss and the regression loss to improve the regression performance of the regression layer and ensure the analysis accuracy of the regression layer.
  • a loss function facilitates the joint learning optimization of the target fusion feature learning loss and the regression loss to improve the regression performance of the regression layer and ensure the analysis accuracy of the regression layer.
  • the deep learning system mainly includes: a clinical data processing module, a computational model building module, a biomarker analysis module, and a clinical application verification module.
  • the clinical data processing module is mainly responsible for the processing and analysis of clinical data, mainly including data sorting, cleaning and preprocessing.
  • the computational model building module is mainly responsible for the construction and training of deep learning networks for predicting drug treatment effects, including: the design, construction, training and testing of deep learning models based on deep learning networks and supervised regression models, and quantitative evaluation of biological sign prediction results.
  • the biomarker analysis module is mainly responsible for the mining and analysis of biological features at the individual level.
  • the biological sign prediction results output by the trained deep learning network are visualized and analyzed to mine and parse the individualized biological features for drug efficacy response, and the model and biological features are replicated and analyzed and the convergence is verified on the data sets of different research sites and different types of data sets.
  • XAI explainable artificial intelligence method
  • the clinical application verification module is mainly responsible for guiding and assisting psychiatrists' clinical treatment through individualized biological characteristics, and optimizing and clinically verifying models and biomarkers to reveal the response phenotype of psychiatric drug treatment, separate psychiatric drug and placebo responses, promote neurobiological understanding of the efficacy of psychiatric drugs, and provide preliminary evidence for potential treatment options.
  • the clinical data processing module, computational model building module, biomarker analysis module and clinical application verification module of the above-mentioned deep learning system can be integrated into a large-scale medical auxiliary treatment intelligent system, or each module can be implemented separately.
  • the clinical data set, drug efficacy prediction deep network and individualized biological characteristics of the above-mentioned deep learning system can be stored in a dedicated server and accessed by other modules through local or cloud services, or integrated with other modules into the same system as a whole.
  • the deep learning system provided in this embodiment is more predictive of the drug efficacy response of mental illness through the deep learning network constructed by objective biological information and big data mining, and deep learning can be applied to automated analysis at the individual level, so that the model constructed based on the deep learning network can effectively assist clinical treatment.
  • the individualized analysis method based on explainable artificial intelligence is more accurate in mining the biological characteristics of highly heterogeneous mental illnesses, and can provide scalable and replicable biomarkers for clinical treatment, which can assist clinicians in formulating corresponding intervention measures and provide effective support for the clinical treatment of mental illness.
  • a multimodal neural bio-signal processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter.
  • the term "module” can implement a combination of software and/or hardware of a predetermined function.
  • the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
  • This embodiment provides a multimodal neural biological signal processing device, which can be used in a deep learning network, the deep learning network includes a pre-built deep learning model and a deep regression model, the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer.
  • the multimodal neural biological signal processing device includes:
  • the acquisition module 41 is used to acquire the multimodal neural biological signal to be processed, preprocess the multimodal neural biological signal, and obtain the preprocessed multimodal neural biological signal.
  • the feature extraction module 42 is used to input the preprocessed multimodal neural biological signal into the deep learning model, and extract the deep features of each modality of the multimodal neural biological signal based on the deep learning model.
  • the feature fusion module 43 is used to input multiple depth features into the feature fusion layer for feature fusion to obtain target fusion features.
  • the prediction module 44 is used to input the target fusion features into the regression layer after passing through the fully connected layer, and use the regression layer to perform physical sign prediction on the target fusion features to generate biological sign prediction results.
  • a plurality of deep learning sub-models are pre-constructed in the deep learning model, and the feature extraction module 42 may include:
  • the type determination submodule is used to determine the type in the multimodal neural biological signal.
  • the sub-model determination sub-module is used to determine each deep learning sub-model corresponding to each type of neural biological signal.
  • the feature extraction submodule is used to input each type of neural biological signal into the corresponding deep learning submodel to obtain the deep features corresponding to each type of neural biological signal.
  • the first deep learning sub-model corresponding to the first type of neural bio-signal includes a model group consisting of multiple first sub-models connected in series and a second sub-model.
  • the above-mentioned feature extraction submodule can be specifically used for: inputting the first type of neural biological signal into the second sub-model for feature extraction to obtain first feature data; fusing the first feature data with the preconfigured position code to obtain fused data; inputting the fused data into the model group for feature extraction to obtain the deep features corresponding to the first type of neural biological signal.
  • the first type of neural biological signal is an electroencephalogram signal
  • the first sub-model includes a Transformer model
  • the second sub-model includes a convolutional neural network model.
  • the convolutional neural network includes a convolutional layer and an average pooling layer.
  • the Transformer model includes an attention module and a feedforward network module.
  • the attention module includes a multi-head attention layer and a first normalization layer
  • the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer.
  • the second deep learning sub-model corresponding to the second type of neural biological signal includes a model group consisting of multiple first sub-models connected in series and a third sub-model, and the third sub-model includes a convolutional layer, a maximum pooling layer and a fully connected layer.
  • the above-mentioned feature extraction submodule can be specifically used to: input the second type of neural biological signal into the convolution layer of the third sub-model for multi-dimensional convolution processing to obtain the convolution processing result; input the convolution processing result into the model group for feature extraction, and pass through the maximum pooling layer and the fully connected layer to output the deep features corresponding to the second type of neural biological signal.
  • the second type of neural biological signal is a functional magnetic resonance signal
  • the first sub-model includes a Transformer model
  • the third sub-model includes a point 4D convolutional network model.
  • the point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer, and a fully connected layer; a model group consisting of multiple Transformer models connected in series is set between the point 4D convolutional layer and the maximum pooling layer;
  • the Transformer model includes an attention module and a feedforward network module;
  • the attention module includes a multi-head attention layer and a first normalization layer;
  • the feedforward network module includes a fully connected feedforward layer and a second normalization layer;
  • the multi-head attention layer is connected to the first normalization layer;
  • the fully connected feedforward layer is connected to the second normalization layer; and the first normalization layer is connected to the fully connected feedforward layer.
  • the multimodal neural bio-signal processing device may further include:
  • the data type acquisition module is used to obtain the data type corresponding to the target fusion feature.
  • the loss function building module is used to build the loss function corresponding to the data type.
  • the optimization module is used to optimize the parameters of the deep regression network using the loss function.
  • the regression layer is deployed in the deep regression network.
  • the above loss function construction module is used to construct a loss function based on data error parameters.
  • the data error parameters include mean absolute error, root mean square error, and median absolute error.
  • the constructed loss function is:
  • Loss ⁇ *RMSE+ ⁇ *MAE+ ⁇ *MedAE+ ⁇ *
  • Loss represents the loss function
  • RMSE represents the root mean square error
  • MAE represents the mean absolute error
  • MedAE represents the median absolute error
  • represents the regularization term
  • ⁇ , ⁇ , ⁇ , and ⁇ represent network parameters.
  • the multimodal neural bio-signal processing device may further include:
  • the numerical range acquisition module is used to obtain the numerical range of the biological sign prediction result.
  • the quantization module is used to quantize the numerical range, divide the numerical range into several intervals, and obtain prediction levels corresponding to the several intervals.
  • the multimodal neural bio-signal processing device may further include:
  • the biometric feature extraction module is used to extract biometric features from the biological sign prediction results based on a preset explainable artificial intelligence method.
  • the analysis module is used to perform replication analysis and convergence analysis on the biometrics and determine individual biomarkers from the biometrics.
  • the multimodal neural bio-signal processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and/or other devices that can provide the above functions.
  • the multimodal neural biological signal processing device uses a deep learning network to extract features from multimodal neural biological signals and fuse multiple deep features, and then predicts the physical signs of the target fused features obtained by fusion to obtain biological feature prediction results. Therefore, the device can fuse multiple deep features based on objective multimodal neural biological signals to achieve effective capture of multiple deep features without artificial design or feature selection, and the deep learning network has a multi-level nonlinear structure, which can effectively analyze the nonlinear relationship in neural biological signals, so that effective biological signs can be analyzed from multiple deep features, achieving effective prediction of biological representations of neural signals and improving the accuracy of biological representation prediction.
  • the embodiment of the present application also provides a server, which may be a server, a computer, etc.
  • the server has a multimodal neural biological signal processing device as shown in FIG. 11 above.
  • the server may include: at least one processor 501, such as a central processing unit (CPU), at least one communication interface 503, a memory 504, and at least one communication bus 502.
  • the communication bus 502 is used to realize the connection and communication between these components.
  • the communication interface 503 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 503 may also include a standard wired interface and a wireless interface.
  • the memory 504 may be a high-speed volatile random access memory (Random Access Memory, RAM) or a non-volatile memory (non-volatile memory), such as at least one disk storage.
  • RAM volatile random access memory
  • non-volatile memory non-volatile memory
  • the memory 504 may also be at least one storage device located away from the aforementioned processor 501.
  • the processor 501 may be combined with the device described in FIG. 11, the memory 504 stores an application program, and the processor 501 calls the program code stored in the memory 504 to perform any of the above method steps.
  • the server can run an operating system stored in the memory 504, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • the present application does not limit the specific form of the operating system.
  • the operating system can be used as a software environment to support the execution of application programs and to manage the server's software and hardware resources.
  • the communication bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus.
  • the communication bus 502 may be divided into an address bus, a data bus, a control bus, etc.
  • FIG12 only uses one thick line, but does not mean that there is only one bus or one type of bus.
  • the memory 504 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); the memory 504 may also include a combination of the above-mentioned types of memory.
  • volatile memory such as a random-access memory (RAM)
  • non-volatile memory such as a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD)
  • flash memory flash memory
  • HDD hard disk drive
  • SSD solid-state drive
  • the memory 504 may also include a combination of the above-mentioned types of memory.
  • processor 501 can be a central processing unit (CPU), a network processor (NP) or a combination of CPU and NP.
  • CPU central processing unit
  • NP network processor
  • the processor 501 may further include a hardware chip.
  • the hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof.
  • the PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
  • the memory 504 is also used to store program instructions.
  • the processor 501 can call the program instructions to implement the multimodal neural bio-signal processing method shown in the above embodiments of the present application.
  • the embodiment of the present application also provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the multimodal neural biological signal processing method in any of the above method embodiments.
  • the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Computation (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Veterinary Medicine (AREA)
  • Psychology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Computational Linguistics (AREA)
  • Medical Informatics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Computing Systems (AREA)
  • Pathology (AREA)
  • Public Health (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Psychiatry (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

A multimodal neural biological signal processing method and apparatus, and a server and a storage medium. The method comprises: acquiring multimodal neural biological signals to be processed, and preprocessing the multimodal neural biological signals (S11); inputting the preprocessed multimodal neural biological signals into a deep learning model, and extracting depth features of each modal of neural biological signals among the multimodal neural biological signals on the basis of the deep learning model (S12); inputting a plurality of kinds of depth features into a feature fusion layer for feature fusion, so as to obtain a target fused feature (S13); and inputting the target fused feature into a regression layer via a fully connected layer, and performing sign prediction on the target fused feature by using the regression layer, so as to generate a biological sign prediction result (S14). By means of implementing the solution, a plurality of kinds of depth features can be effectively captured without the need for artificially designing or selecting the features; moreover, effective prediction of the biological representation of a neural signal is realized, thereby improving the accuracy of prediction of biological representation.

Description

多模态神经生物信号处理方法、装置、服务器及存储介质Multimodal neural biological signal processing method, device, server and storage medium 技术领域Technical Field
本申请涉及数据处理技术领域，具体涉及多模态神经生物信号的处理方法、装置、服务器及存储介质。The present application relates to the field of data processing technology, and specifically to a method, device, server and storage medium for processing multimodal neural biological signals.
背景技术Background technique
随着精神疾病的治疗药物的广泛使用，对于精神药物的多种神经生物表征受到越来越多的关注。由于缺乏不同数据集的交叉验证和数据样本量小，目前，仍然比较缺乏精神疾病治疗药物应答表型的强大的神经生物学特征。With the widespread use of drugs for the treatment of mental illness, the various neurobiological characterizations of psychotropic drugs have received increasing attention. However, due to the lack of cross-validation of different datasets and small sample sizes, there is still a lack of strong neurobiological characteristics of the drug response phenotype for the treatment of mental illness.
目前，通常采用经典的机器学习方法用于解析rsEEG、磁共振(MRI)等神经生物数据中存在的复杂多变关系，以识别药物治疗后的神经生物表征。尽管通过经典的机器学习计算模型能够在一定程度上反映出神经生物特征，但其需要依赖人为经验设计或选取特征，难以有效的解析出神经生物信号中的复杂的非线性关系以确定治疗药物应答表型的神经生物学特征，存在一定的局限性，从而影响了对于神经生物表征的预测精度。因此，如何能够实现对于神经生物表征的精准预测仍亟待解决。At present, classical machine learning methods are usually used to analyze the complex and variable relationships in neurobiological data such as rsEEG and magnetic resonance imaging (MRI) to identify neurobiological representations after drug treatment. Although the classical machine learning computational model can reflect neurobiological characteristics to a certain extent, it relies on artificial experience to design or select features. It is difficult to effectively analyze the complex nonlinear relationships in neurobiological signals to determine the neurobiological characteristics of the therapeutic drug response phenotype. There are certain limitations, which affects the prediction accuracy of neurobiological representations. Therefore, how to achieve accurate prediction of neurobiological representations remains to be solved.
发明内容Summary of the invention
有鉴于此，本申请实施例提供了多模态神经生物信号的处理方法、装置、服务器及存储介质，以解决神经生物学特征难于精准预测的问题。In view of this, the embodiments of the present application provide a method, device, server and storage medium for processing multimodal neurobiological signals to solve the problem that neurobiological characteristics are difficult to accurately predict.
根据第一方面，本申请实施例提供的多模态神经生物信号的处理方法，用于深度学习网络中，所述深度学习网络包括预先构建的深度学习模型、深度回归模型，所述深度回归模型包括特征融合层、全连接层和回归层，所述方法包括：获取待处理的多模态神经生物信号，对所述多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号；将所述预处理后的多模态神经生物信号输入至所述深度学习模型中，基于所述深度学习模型提取所述多模态神经生物信号中每种模态神经生物信号的深度特征；将多种所述深度特征输入至所述特征融合层进行特征融合，得到目标融合特征；将所述目标融合特征经由所述全连接层后输入至所述回归层，利用所述回归层对所述目标融合特征进行体征预测，生成生物体征预测结果。According to the first aspect, the method for processing multimodal neural biological signals provided in the embodiment of the present application is used in a deep learning network, wherein the deep learning network includes a pre-built deep learning model and a deep regression model, wherein the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer, and the method includes: obtaining a multimodal neural biological signal to be processed, preprocessing the multimodal neural biological signal to obtain a preprocessed multimodal neural biological signal; inputting the preprocessed multimodal neural biological signal into the deep learning model, and extracting deep features of each modality of neural biological signal in the multimodal neural biological signal based on the deep learning model; inputting a plurality of the deep features into the feature fusion layer for feature fusion to obtain a target fusion feature; inputting the target fusion feature into the regression layer after passing through the fully connected layer, and using the regression layer to perform sign prediction on the target fusion feature to generate a biological sign prediction result.
本申请实施例提供的多模态神经生物信号的处理方法，采用深度学习网络对多模态神经生物信号进行特征提取以及多种深度特征的融合，继而对融合得到的目标融合特征进行体征预测，得到生物特征预测结果。由此能够基于客观的多模态神经生物信号对多种深度特征进行融合，以实现对于多种深度特征的有效捕捉，无需人为设计或选取特征，且深度学习网络具有多层次非线性结构，能够有效解析神经生物信号中所存在的非线性关系，从而能够从多种深度特征中解析出有效的生物体征，实现了对于神经信号的生物表征的有效预测，提高了生物表征的预测准确度。The method for processing multimodal neural biological signals provided in the embodiment of the present application uses a deep learning network to extract features from multimodal neural biological signals and fuse multiple deep features, and then predicts signs of the target fused features obtained by fusion to obtain biological feature prediction results. In this way, multiple deep features can be fused based on objective multimodal neural biological signals to achieve effective capture of multiple deep features without artificial design or feature selection, and the deep learning network has a multi-level nonlinear structure, which can effectively analyze the nonlinear relationship in neural biological signals, so that effective biological signs can be analyzed from multiple deep features, achieving effective prediction of biological representations of neural signals and improving the accuracy of biological representation prediction.
结合第一方面，在第一方面的第一实施方式中，所述深度学习模型中构建有多个深度学习子模型；将所述多模态神经生物信号输入至所述深度学习模型中，基于所述深度学习模型提取所述多模态神经生物信号中每种模态神经生物信号的深度特征，包括：确定多模态神经生物信号中的类型；确定与各个类型的神经生物信号所对应的各个深度学习子模型；分别将所述各个类型的神经生物信号输入至相应的深度学习子模型中，得到与所述各个类型的神经生物信号相对应的深度特征。In combination with the first aspect, in a first implementation of the first aspect, a plurality of deep learning sub-models are constructed in the deep learning model; the multimodal neural biological signal is input into the deep learning model, and the deep features of each modality of the neural biological signal in the multimodal neural biological signal are extracted based on the deep learning model, including: determining the type of the multimodal neural biological signal; determining the deep learning sub-models corresponding to each type of neural biological signal; and inputting each type of neural biological signal into the corresponding deep learning sub-model respectively to obtain the deep features corresponding to each type of neural biological signal.
本申请实施例提供的多模态神经生物信号的处理方法，通过将不同类型的神经生物信号输入至相应的深度学习子模型以进行深度特征提取，便于定向提取各个类型的神经生物信号的深度特征，提高了深度特征的提取精度。The multimodal neural biological signal processing method provided in the embodiment of the present application facilitates the directional extraction of deep features of each type of neural biological signal by inputting different types of neural biological signals into corresponding deep learning sub-models for deep feature extraction, thereby improving the extraction accuracy of deep features.
结合第一方面第一实施方式，在第一方面的第二实施方式中，当所述神经生物信号为第一类型神经生物信号时，与所述第一类型神经生物信号对应的第一深度学习子模型包括多个第一子模型串联组成的模 型组以及第二子模型；将所述第一类型神经生物信号输入至所述第一深度学习子模型中，得到与所述第一类型神经生物信号相对应的深度特征，包括：将所述第一类型神经生物信号输入至第二子模型进行特征提取，得到第一特征数据；将所述第一特征数据与预配置的位置编码进行融合，得到融合数据；将所述融合数据输入至所述模型组进行特征提取，得到所述第一类型神经生物信号相对应的深度特征。In combination with the first embodiment of the first aspect, in the second embodiment of the first aspect, when the neural biological signal is a first type of neural biological signal, the first deep learning sub-model corresponding to the first type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a second sub-model; the first type of neural biological signal is input into the first deep learning sub-model to obtain a deep feature corresponding to the first type of neural biological signal, including: inputting the first type of neural biological signal into the second sub-model for feature extraction to obtain first feature data; fusing the first feature data with a preconfigured position code to obtain fused data; inputting the fused data into the model group for feature extraction to obtain a deep feature corresponding to the first type of neural biological signal.
结合第一方面第二实施方式，在第一方面的第三实施方式中，所述第一类型神经生物信号为脑电信号；所述第一子模型包括Transformer模型；所述第二子模型包括卷积神经网络模型；所述卷积神经网络包括卷积层以及平均池化层；所述Transformer模型包括注意力模块和前馈网络模块；所述注意力模块包括多头注意力层以及第一归一化层；所述前馈网络模块包括全连接前馈层以及第二归一化层；所述多头注意力层与所述第一归一化层连接；所述全连接前馈层与所述第二归一化层连接；所述第一归一化层与所述全连接前馈层连接；所述注意力模块和所述前馈网络模块具有对应的快捷连接。In combination with the second embodiment of the first aspect, in the third embodiment of the first aspect, the first type of neural biological signal is an electroencephalogram signal; the first sub-model includes a Transformer model; the second sub-model includes a convolutional neural network model; the convolutional neural network includes a convolutional layer and an average pooling layer; the Transformer model includes an attention module and a feedforward network module; the attention module includes a multi-head attention layer and a first normalization layer; the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer; the attention module and the feedforward network module have corresponding shortcut connections.
本申请实施例提供的多模态神经生物信号的处理方法，通过构建针对于脑电信号的第一深度学习子模型，且该第一深度学习子模型由多层深度结构(第一子模型和多个第二子模型组成的模型组)组成的，由此能够对脑电信号进行高维度的特征提取，从而能够更加准确的挖掘和提取脑电信号的时空特征以及长程依赖关系。The method for processing multimodal neural biological signals provided in the embodiment of the present application constructs a first deep learning sub-model for EEG signals, and the first deep learning sub-model is composed of a multi-layer deep structure (a model group consisting of a first sub-model and multiple second sub-models), thereby being able to perform high-dimensional feature extraction on EEG signals, thereby being able to more accurately mine and extract the spatiotemporal characteristics and long-range dependencies of EEG signals.
结合第一方面第一实施方式，在第一方面的第四实施方式中，当所述神经生物信号为第二类型神经生物信号时，与所述第二类型神经生物信号对应的第二深度学习子模型包括多个第一子模型串联组成的模型组以及第三子模型，所述第三子模型包括卷积层、最大池化层及全连接层；将所述第二类型神经生物信号输入至第二深度学习子模型中，得到与所述第二类型神经生物信号相对应的深度特征，包括：将所述第二类型神经生物信号输入至所述第三子模型的卷积层进行多维卷积处理，得到卷积处理结果；将所述卷积处理结果输入至所述模型组进行特征提取，并经过所述最大池化层与所述全连接层，输出所述第二类型神经生物信号相对应的深度特征。In combination with the first embodiment of the first aspect, in the fourth embodiment of the first aspect, when the neural biological signal is a second type of neural biological signal, the second deep learning sub-model corresponding to the second type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a third sub-model, and the third sub-model includes a convolution layer, a maximum pooling layer and a fully connected layer; the second type of neural biological signal is input into the second deep learning sub-model to obtain a deep feature corresponding to the second type of neural biological signal, including: inputting the second type of neural biological signal into the convolution layer of the third sub-model for multi-dimensional convolution processing to obtain a convolution processing result; inputting the convolution processing result into the model group for feature extraction, and passing through the maximum pooling layer and the fully connected layer, outputting the deep feature corresponding to the second type of neural biological signal.
结合第一方面第四实施方式，在第一方面的第五实施方式中，所述第二类型神经生物信号为功能磁共振信号；所述第一子模型包括Transformer模型；所述第三子模型包括点4D卷积网络模型；所述点4D卷积网络模型包括点4D卷积层、最大池化层及全连接层；多个所述Transformer模型串联组成的模型组设置在所述点4D卷积层与最大池化层之间；所述Transformer模型包括注意力模块和前馈网络模块；所述注意力模块包括多头注意力层以及第一归一化层；所述前馈网络模块包括全连接前馈层以及第二归一化层；所述多头注意力层与所述第一归一化层连接；所述全连接前馈层与所述第二归一化层连接；所述第一归一化层与所述全连接前馈层连接；；所述注意力模块和所述前馈网络模块具有对应的快捷连接。In combination with the fourth embodiment of the first aspect, in the fifth embodiment of the first aspect, the second type of neural biological signal is a functional magnetic resonance signal; the first sub-model includes a Transformer model; the third sub-model includes a point 4D convolutional network model; the point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer and a fully connected layer; a model group composed of multiple Transformer models connected in series is arranged between the point 4D convolutional layer and the maximum pooling layer; the Transformer model includes an attention module and a feedforward network module; the attention module includes a multi-head attention layer and a first normalization layer; the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer; the attention module and the feedforward network module have corresponding shortcut connections.
本申请实施例提供的多模态神经生物信号的处理方法，通过构建针对于功能磁共振信号的第二深度学习子模型，且该第二深度学习子模型由多层深度结构(第三子模型和多个第二子模型组成的模型组)组成的，由此实现了对于功能磁共振信号的高维度特征提取，从而能够更加准确的挖掘和提取功能磁共振信号的空间特征以及功能连接特征。The method for processing multimodal neurobiological signals provided in the embodiment of the present application constructs a second deep learning sub-model for functional magnetic resonance signals, and the second deep learning sub-model is composed of a multi-layer deep structure (a model group consisting of a third sub-model and multiple second sub-models), thereby realizing high-dimensional feature extraction of functional magnetic resonance signals, thereby being able to more accurately mine and extract the spatial features and functional connectivity features of functional magnetic resonance signals.
结合第一方面，在第一方面的第六实施方式中，所述将所述目标融合特征经由所述全连接层后输入至所述回归层，利用所述回归层对所述目标融合特征进行体征预测，生成生物体征预测结果之前，还包括：获取所述目标融合特征对应的数据类型；构建与所述数据类型对应的损失函数；利用所述损失函数对深度回归网络的参数进行优化处理，所述回归层部署在所述深度回归网络中。In combination with the first aspect, in a sixth implementation of the first aspect, the target fusion feature is input into the regression layer after passing through the fully connected layer, and the target fusion feature is predicted by the regression layer, before generating a biological sign prediction result, it also includes: obtaining the data type corresponding to the target fusion feature; constructing a loss function corresponding to the data type; using the loss function to optimize the parameters of the deep regression network, and the regression layer is deployed in the deep regression network.
本申请实施例提供的多模态神经生物信号的处理方法，通过构建损失函数，便于将目标融合特征学习损失与回归损失进行联合学习优化，以提高回归层的回归性能，保证回归层的分析精度。The multimodal neural biological signal processing method provided in the embodiment of the present application facilitates the joint learning optimization of the target fusion feature learning loss and the regression loss by constructing a loss function, so as to improve the regression performance of the regression layer and ensure the analysis accuracy of the regression layer.
结合第一方面第六实施方式，在第一方面的第七实施方式中，所述构建与所述数据类型对应的损失函数，包括：当所述数据类型为连续类型时，基于数据误差参数构建损失函数；其中，所述数据误差参数包括平均绝对误差、均方根误差以及中位绝对误差，所述损失函数为：Loss＝α*RMSE+β*MAE+μ*MedAE+λ*||W||；其中，Loss表示损失函数；RMSE表示均方根误差；MAE表示平均绝对误差；MedAE表示中位绝对误差；||W||表示正则化项；α、β、μ、λ表示网络参数。In combination with the sixth implementation of the first aspect, in the seventh implementation of the first aspect, constructing a loss function corresponding to the data type includes: when the data type is a continuous type, constructing a loss function based on data error parameters; wherein the data error parameters include mean absolute error, root mean square error and median absolute error, and the loss function is: Loss = α*RMSE+β*MAE+μ*MedAE+λ*||W||; wherein, Loss represents the loss function; RMSE represents the root mean square error; MAE represents the mean absolute error; MedAE represents the median absolute error; ||W|| represents the regularization term; α, β, μ, λ represent network parameters.
本申请实施例提供的多模态神经生物信号的处理方法，在构建损失函数时添加正则化项以对损失函 数进行修正，由此能够最大程度的降低过拟合。The method for processing multimodal neural biological signals provided in the embodiment of the present application adds a regularization term when constructing the loss function to correct the loss function, thereby minimizing overfitting.
结合第一方面，在第一方面的第八实施方式中，所述方法还包括：获取所述生物体征预测结果所处的数值范围；对所述数值范围进行量化处理，将所述数值范围划分为若干个区间，得到对应于所述若干个区间的预测级别。In combination with the first aspect, in the eighth implementation of the first aspect, the method also includes: obtaining the numerical range of the biological sign prediction result; quantizing the numerical range, dividing the numerical range into a number of intervals, and obtaining prediction levels corresponding to the several intervals.
本申请实施例提供的多模态神经生物信号的处理方法，通过对数值范围进行量化处理以将其划分为若干个区间，并为不同的区间设定不同的预测级别，能够辅助临床医生进行生物表征的有效评估。The multimodal neural biological signal processing method provided in the embodiment of the present application can assist clinicians in effectively evaluating biological representations by quantifying the numerical range to divide it into several intervals and setting different prediction levels for different intervals.
结合第一方面，在第一方面的第九实施方式中，所述方法还包括：基于预设的可解释人工智能方法从所述生物体征预测结果中提取生物特征；对所述生物特征进行复制分析和收敛性分析，从所述生物特征中确定出个体生物标记。In combination with the first aspect, in a ninth implementation of the first aspect, the method further includes: extracting biological features from the biological sign prediction results based on a preset explainable artificial intelligence method; performing replication analysis and convergence analysis on the biological features, and determining individual biomarkers from the biological features.
本申请实施例提供的多模态神经生物信号的处理方法，通过可解释人工智能应用于生物体征预测结果进行生物特征的提取，并通过复制分析和收敛性分析以确定出个体化生物标记，对于生物特征的可移植性和稳定性具有较大的神经生物学意义，能够为面对精神疾病的临床医生提供潜在的有力辅助。The multimodal neurobiological signal processing method provided in the embodiment of the present application applies explainable artificial intelligence to the biological sign prediction results to extract biological characteristics, and determines individualized biomarkers through replication analysis and convergence analysis. It has great neurobiological significance for the portability and stability of biological characteristics, and can provide potential powerful assistance to clinicians facing mental illness.
根据第二方面，本申请实施例提供了一种多模态神经生物信号的处理装置，用于深度学习网络中，所述深度学习网络包括预先构建的深度学习模型、深度回归模型，所述深度回归模型包括特征融合层、全连接层和回归层，所述装置包括：获取模块，用于获取待处理的多模态神经生物信号，对所述多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号；特征提取模块，用于将所述预处理后的多模态神经生物信号输入至所述深度学习模型中，基于所述深度学习模型提取所述多模态神经生物信号中每种模态神经生物信号的深度特征；特征融合模块，用于将多种所述深度特征输入至所述特征融合层进行特征融合，得到目标融合特征；预测模块，用于将所述目标融合特征经由所述全连接层后输入至所述回归层，利用所述回归层对所述目标融合特征进行体征预测，生成生物体征预测结果。According to the second aspect, an embodiment of the present application provides a multimodal neural biological signal processing device for use in a deep learning network, wherein the deep learning network includes a pre-built deep learning model and a deep regression model, wherein the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer, and the device includes: an acquisition module, used to acquire the multimodal neural biological signal to be processed, pre-process the multimodal neural biological signal, and obtain the pre-processed multimodal neural biological signal; a feature extraction module, used to input the pre-processed multimodal neural biological signal into the deep learning model, and extract the deep features of each modality of the multimodal neural biological signal in the multimodal neural biological signal based on the deep learning model; a feature fusion module, used to input a plurality of the deep features into the feature fusion layer for feature fusion, and obtain a target fusion feature; a prediction module, used to input the target fusion feature into the regression layer after passing through the fully connected layer, and use the regression layer to perform sign prediction on the target fusion feature to generate a biological sign prediction result.
根据第三方面，本申请实施例提供了一种服务器，包括：存储器和处理器，所述存储器和所述处理器之间互相通信连接，所述存储器中存储有计算机指令，所述处理器通过执行所述计算机指令，从而执行第一方面或第一方面任一实施方式所述的多模态神经生物信号的处理方法。According to the third aspect, an embodiment of the present application provides a server, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the method for processing multimodal neural biological signals described in the first aspect or any embodiment of the first aspect.
根据第四方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使计算机执行第一方面或第一方面任一实施方式所述的多模态神经生物信号的处理方法。According to the fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method for processing multimodal neural biological signals described in the first aspect or any embodiment of the first aspect.
需要说明的是，本申请实施例提供的多模态神经生物信号的处理装置、服务器以及计算机可读存储介质的相应有益效果，请参见多模态神经生物信号的处理方法中相应内容的描述，在此不再赘述。It should be noted that the corresponding beneficial effects of the multimodal neural biological signal processing device, server and computer-readable storage medium provided in the embodiments of the present application can be found in the description of the corresponding contents in the multimodal neural biological signal processing method, which will not be repeated here.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更清楚地说明本申请具体实施方式或现有技术中的技术方案，下面将对具体实施方式或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
图1是本申请一实施例的多模态神经生物信号的处理方法的流程图；FIG1 is a flow chart of a method for processing multimodal neural biosignals according to an embodiment of the present application;
图2是本申请一实施例的深度学习网络的示意图；FIG2 is a schematic diagram of a deep learning network according to an embodiment of the present application;
图3是本申请一实施例的多模态神经生物信号的处理方法的另一流程图；FIG3 is another flow chart of a method for processing multimodal neural biosignals according to an embodiment of the present application;
图4是本申请一实施例的深度学习网络的另一示意图；FIG4 is another schematic diagram of a deep learning network according to an embodiment of the present application;
图5是本申请一实施例的CNN-Trans模型的融合示意图；FIG5 is a schematic diagram of the fusion of a CNN-Trans model according to an embodiment of the present application;
图6是本申请一实施例的4D-Trans模型的融合示意图；FIG6 is a schematic diagram of the fusion of a 4D-Trans model according to an embodiment of the present application;
图7是本申请一实施例的多模态神经生物信号的处理方法的又一流程图；FIG7 is another flow chart of a method for processing multimodal neural biosignals according to an embodiment of the present application;
图8是本申请一实施例的生物特征的确定示意图；FIG8 is a schematic diagram of determining a biometric feature according to an embodiment of the present application;
图9是本申请一实施例的临床辅助应用的示意图；FIG9 is a schematic diagram of a clinical auxiliary application according to an embodiment of the present application;
图10是本申请一实施例的深度学习***的示意图；FIG10 is a schematic diagram of a deep learning system according to an embodiment of the present application;
图11是本申请一实施例的多模态神经生物信号的处理装置的结构框图；FIG11 is a structural block diagram of a multimodal neural bio-signal processing device according to an embodiment of the present application;
图12是本申请一实施例提供的服务器的硬件结构示意图。FIG. 12 is a schematic diagram of the hardware structure of a server provided in an embodiment of the present application.
具体实施方式Detailed ways
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
根据本申请实施例，提供了一种多模态神经生物信号的处理方法的实施例，需要说明的是，在附图的流程图示出的步骤可以在诸如一组计算机可执行指令的计算机***中执行，并且，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。According to an embodiment of the present application, an embodiment of a method for processing a multimodal neurobiological signal is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
在本实施例中提供了一种多模态神经生物信号的处理方法，可用于深度学习网络中，该深度学习网络包括预先构建的深度学习模型、深度回归模型，深度回归模型包括特征融合层、全连接层和回归层。In this embodiment, a method for processing multimodal neural biological signals is provided, which can be used in a deep learning network. The deep learning network includes a pre-built deep learning model and a deep regression model. The deep regression model includes a feature fusion layer, a fully connected layer and a regression layer.
图1是根据本申请实施例的多模态神经生物信号的处理方法的流程图，如图1所示，该流程包括如下步骤：FIG. 1 is a flow chart of a method for processing a multimodal neural biosignal according to an embodiment of the present application. As shown in FIG. 1 , the process includes the following steps:
S11，获取待处理的多模态神经生物信号，对多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号。S11, obtaining a multimodal neural biological signal to be processed, preprocessing the multimodal neural biological signal, and obtaining a preprocessed multimodal neural biological signal.
多模态神经生物信号为通过采集设备针对于精神疾病的人体所采集到的多种类型的神经生物信号。多模态神经生物学信号可以从临床数据库中获取，该数据库中包含有多个研究站点针对精神疾病患者所构建的神经生物信号数据集。Multimodal neurobiological signals are various types of neurobiological signals collected by acquisition equipment from people with mental illness. Multimodal neurobiological signals can be obtained from clinical databases, which contain neurobiological signal datasets constructed by multiple research sites for patients with mental illness.
具体地，多模态神经生物信号可以包括脑电信号(EEG)、功能磁共振信号(fMRI)以及行为量表数据等。Specifically, the multimodal neurobiological signals may include electroencephalogram (EEG) signals, functional magnetic resonance imaging (fMRI) signals, and behavioral scale data.
由于多模态神经生物信号的采集过程中难以避免因采集设备本身、病患本身或外界干扰而引入伪迹，因此在对多模态神经生物信号进行分析之前，需要对其进行预处理。Since it is difficult to avoid the introduction of artifacts due to the acquisition equipment itself, the patient itself or external interference during the acquisition of multimodal neural biological signals, it is necessary to preprocess the multimodal neural biological signals before analyzing them.
对于EEG而言采用全自动的伪迹剔除流程，以尽可能最小化人工去除伪迹在预处理过程中导致的偏差。具体的预处理步骤包括：For EEG, a fully automatic artifact removal process is used to minimize the deviation caused by manual artifact removal during preprocessing. The specific preprocessing steps include:
a)对EEG进行降采样处理，以将其降采样至250Hz。a) Downsampling the EEG to 250 Hz.
b)采用陷波滤波器消除EEG中所存在的50Hz的工频干扰。b) Use a notch filter to eliminate the 50 Hz power frequency interference in the EEG.
c)用0.01Hz的高通滤波器去除EEG中的非生理慢波漂移。c) A 0.01 Hz high-pass filter was used to remove non-physiological slow wave drifts in the EEG.
d)采用全脑平均法对EEG数据进行重参考。d) EEG data were re-referenced using whole-brain averaging.
e)按时间维度对EEG进行分段，例如每段EEG的长度为3s。e) Segment the EEG according to the time dimension, for example, the length of each EEG segment is 3s.
f)采用波幅阈值方法剔除伪迹段，即通过检测EEG的波幅是否超过设定的阈值来进行伪迹段的剔除。f) The amplitude threshold method is used to remove the artifact segment, that is, the artifact segment is removed by detecting whether the EEG amplitude exceeds the set threshold.
g)通过对通道间空间相关性进行阈值处理以剔除坏通道，利用插值法从相邻通道的脑电中插值出坏通道的EEG。由于正常通道的EEG与相邻通道的EEG具有一定的相关性，但坏通道的EEG与相邻通道的EEG不相关或者相关性极低，通过计算相邻通道间的EEG的相关性即可实现坏通道的剔除。例如，计算相邻通道之间的皮尔逊相关系数，当皮尔逊相关系数低于阈值时，表示存在坏通道并进行剔除。由于坏通道的EEG可能是由于采集过程引入了干扰和噪声而导致的不可用，若直接丢弃坏通道的EEG则会导致EEG的导联数减少而数量不够，此时可以基于正常通道的EEG与相邻通道的EEG之间的相关性，采用插值法(例如球面样条插值、三次样条插值等)模拟出坏通道的EEG。g) Threshold processing is performed on the spatial correlation between channels to remove bad channels, and the EEG of the bad channel is interpolated from the EEG of the adjacent channels using the interpolation method. Since the EEG of the normal channel has a certain correlation with the EEG of the adjacent channel, but the EEG of the bad channel is not correlated with the EEG of the adjacent channel or the correlation is extremely low, the bad channel can be removed by calculating the correlation of the EEG between adjacent channels. For example, the Pearson correlation coefficient between adjacent channels is calculated. When the Pearson correlation coefficient is lower than the threshold, it indicates that there is a bad channel and it is removed. Since the EEG of the bad channel may be unavailable due to interference and noise introduced during the acquisition process, if the EEG of the bad channel is directly discarded, the number of EEG leads will be reduced and insufficient. At this time, based on the correlation between the EEG of the normal channel and the EEG of the adjacent channel, the EEG of the bad channel can be simulated by using the interpolation method (such as spherical spline interpolation, cubic spline interpolation, etc.).
h)丢弃坏通道率超过20％的受试者的EEG数据。h) Discard the EEG data of subjects with bad channel rate exceeding 20%.
i)使用独立成分分析方法(Independent Component Correlation Algorithm，ICA)识别并剔除肌电、眼电和心电等伪迹。i) Use Independent Component Correlation Algorithm (ICA) to identify and remove artifacts such as electromyography, electrooculography and electrocardiography.
对于fMRI而言，可以基于FSL工具进行预处理，具体步骤包括：For fMRI, preprocessing can be performed based on the FSL tool. The specific steps include:
a)使用仿射配准矩阵和基于边界的配准，将fMRI对应功能性图像重新配准到结构像，将磁共振T1像与MNI标准脑的磁共振T1像的标准模板进行非线性归一化连接。a) Using the affine registration matrix and boundary-based registration, the fMRI corresponding functional images are realigned to the structural images, and the MRI T1 images are nonlinearly normalized and connected to the standard template of the MRI T1 images of the MNI standard brain.
b)从运动校正的功能像中回归与分割出的白质和脑脊液相对应的干扰信号。磁共振扫描中，受试者者在获取图像的过程中可能会挪动身体，这会导致磁共振扫描得到的图像因位置不匹配而存在运动伪影，该运动伪影可以通过刚体变换进行纠正，也可以基于运动校正的AIR软件进行纠正。b) Regression of interference signals corresponding to the segmented white matter and cerebrospinal fluid from the motion-corrected functional image. During MRI scanning, the subject may move his body during the image acquisition process, which will cause motion artifacts in the MRI scanned image due to position mismatch. The motion artifacts can be corrected by rigid body transformation or by motion-corrected AIR software.
通过FSL工具可以直接用来分割出白质、灰质和脑脊液等脑组织类型，然后通过回归算法(例如，自动回归移动平均模型ARMA)从运动校正的功能像中回归出白质与脑脊液对应的干扰信号。The FSL tool can be used to directly segment brain tissue types such as white matter, gray matter, and cerebrospinal fluid, and then the interference signals corresponding to white matter and cerebrospinal fluid are regressed from the motion-corrected functional image through a regression algorithm (for example, the automatic regressive moving average model ARMA).
c)采用6mm半高全宽高斯核平滑fMRI影像。c) fMRI images were smoothed using a 6 mm full width at half maximum Gaussian kernel.
d)设置绝对运动水平的截止点以确保影像测量的质量。该截止点的设置可以根据实际使用场景确定，也支持默认值的选择。d) Set the cutoff point of absolute motion level to ensure the quality of image measurement. The cutoff point can be set according to the actual usage scenario, and the default value can also be selected.
对于行为量表数据而言，行为量表数据包括：年龄、心率、血压、性别、种族及量表评估数据等。对行为量表数据进行预处理，即是对行为量表数据进行数字化及量化处理，生成相应的特征矩阵。For the behavior scale data, the behavior scale data includes: age, heart rate, blood pressure, gender, race and scale assessment data, etc. Preprocessing the behavior scale data means digitizing and quantifying the behavior scale data to generate a corresponding feature matrix.
上述多模态神经生物信号不限于EEG、fMRI以及行为量表数据，还可以包括脑磁图(MEG)、近红外光谱(NIRS)、弥散张量成像(DTI)、正电子发射型计算机断层成像(PET)、眼动、血样、基因等数据，此处不作具体限定，本领域技术人员可以根据实际需求确定。The above-mentioned multimodal neurobiological signals are not limited to EEG, fMRI and behavioral scale data, but may also include magnetoencephalography (MEG), near-infrared spectroscopy (NIRS), diffusion tensor imaging (DTI), positron emission tomography (PET), eye movement, blood samples, genes and other data. No specific limitation is made here, and technical personnel in this field can determine according to actual needs.
需要说明的是，上述多模态神经生物信号包括精神疾病药物与安慰剂的对照数据。It should be noted that the above-mentioned multimodal neurobiological signals include control data of psychiatric drugs and placebos.
S12，将预处理后的多模态神经生物信号输入至深度学习模型中，基于深度学习模型提取多模态神经生物信号中每种模态神经生物信号的深度特征。S12, inputting the preprocessed multimodal neural biological signal into a deep learning model, and extracting the deep features of each modality of the multimodal neural biological signal based on the deep learning model.
深度学习模型为通过多模态神经生物信号预先训练所得到的模型，该深度学习模型以多模态神经生物信号作为输入，以深度特征作为输出，该深度学习模型由多层深度结构组成。将经过预处理的多模态神经生物信号输入至深度学习模型中，通过深度学习模型提取针对于每种模态的神经生物信号的深度特征。The deep learning model is a model obtained by pre-training multimodal neural biological signals. The deep learning model takes multimodal neural biological signals as input and outputs deep features. The deep learning model consists of a multi-layer deep structure. The pre-processed multimodal neural biological signals are input into the deep learning model, and the deep features of the neural biological signals for each modality are extracted through the deep learning model.
S13，将多种深度特征输入至特征融合层进行特征融合，得到目标融合特征。S13, inputting multiple depth features into the feature fusion layer for feature fusion to obtain target fusion features.
将从多模态神经生物信号中提取出的多种深度特征输入至深度回归模型中的特征融合层，通过特征融合层将多种深度特征进行融合，得到经过融合的目标融合特征。The multiple deep features extracted from the multimodal neural biological signals are input into the feature fusion layer in the deep regression model, and the multiple deep features are fused through the feature fusion layer to obtain the fused target fusion features.
S14，将目标融合特征经由全连接层后输入至回归层，利用回归层对目标融合特征进行体征预测，生成生物体征预测结果。S14, inputting the target fusion features into the regression layer after passing through the fully connected layer, and using the regression layer to perform physical sign prediction on the target fusion features to generate biological sign prediction results.
生物体征预测结果用于表征精神疾病药物治疗前、后的预测结果，该生物体征预测结果通过预测标签label来表征，该预测标签label该生物体征预测结果根据精神疾病药物治疗前、后的体征量表数值差计算得到。The biological sign prediction result is used to characterize the prediction results before and after the drug treatment of mental illness. The biological sign prediction result is characterized by a prediction label label, and the prediction label label is calculated based on the difference in the sign scale values before and after the drug treatment of mental illness.
特征融合层通过两层全连接层与回归层连接，特征融合层输出的目标融合特征经过两层全连接层后输入至深度回归模型的回归层中以获取到生物体征预测结果，如图2所示。深度回归模型为有监督学习模型，在回归层中对目标融合特征进行体征预测，并输出生物体征预测结果。The feature fusion layer is connected to the regression layer through two fully connected layers. The target fusion features output by the feature fusion layer are input into the regression layer of the deep regression model after passing through two fully connected layers to obtain the biological sign prediction results, as shown in Figure 2. The deep regression model is a supervised learning model. In the regression layer, the target fusion features are used to predict biological signs and the biological sign prediction results are output.
以HAMD 17为例，预先采集每个受试者服用精神疾病药物前的体征量表HAMD 17数据值H0，用药8周后会再次采集每个受试者的HAMD 17数据H8，将用药前的HAMD 17值减去用药八周后的HAMD 17值(即H0-H8)就是生物体征预测结果。由此能够将多模态神经生物信号的深度特征与精神疾病药效进行关联，实现了对于精神疾病药效的准确预测，并输出相应的生物体征预测结果，以便对预测结果进行评估。 Taking HAMD 17 as an example, the HAMD 17 data value H0 of each subject's physical sign scale before taking psychiatric drugs is collected in advance. After 8 weeks of medication, the HAMD 17 data H8 of each subject will be collected again. The HAMD 17 value before medication minus the HAMD 17 value after 8 weeks of medication (i.e. H0-H8) is the biological sign prediction result. In this way, the deep features of multimodal neural biological signals can be associated with the efficacy of psychiatric drugs, achieving accurate prediction of the efficacy of psychiatric drugs, and outputting the corresponding biological sign prediction results so that the prediction results can be evaluated.
本实施例提供的多模态神经生物信号的处理方法，采用深度学习网络对多模态神经生物信号进行特征提取以及多种深度特征的融合，继而对融合得到的目标融合特征进行体征预测，得到生物特征预测结果。由此能够基于客观的多模态神经生物信号对多种深度特征进行融合，以实现对于多种深度特征的有效捕捉，无需人为设计或选取特征，且深度学习网络具有多层次非线性结构，能够有效解析神经生物信号中所存在的非线性关系，从而能够从多种深度特征中解析出有效的生物体征，实现了对于神经信号的生物表征的有效预测，提高了生物表征的预测准确度。The method for processing multimodal neural biological signals provided in this embodiment uses a deep learning network to extract features from multimodal neural biological signals and fuse multiple deep features, and then predicts the physical signs of the target fused features obtained by fusion to obtain biological feature prediction results. In this way, multiple deep features can be fused based on objective multimodal neural biological signals to achieve effective capture of multiple deep features without artificial design or feature selection. The deep learning network has a multi-level nonlinear structure and can effectively analyze the nonlinear relationship in neural biological signals, so that effective biological signs can be analyzed from multiple deep features, achieving effective prediction of biological representations of neural signals and improving the accuracy of biological representation prediction.
在本实施例中提供了一种多模态神经生物信号的处理方法，可用于深度学习网络中，该深度学习网络包括预先构建的深度学习模型、深度回归模型，深度回归模型包括特征融合层、全连接层和回归层。In this embodiment, a method for processing multimodal neural biological signals is provided, which can be used in a deep learning network. The deep learning network includes a pre-built deep learning model and a deep regression model. The deep regression model includes a feature fusion layer, a fully connected layer and a regression layer.
图3是根据本申请实施例的多模态神经生物信号的处理方法的流程图，如图3所示，该流程包括如下步骤：FIG3 is a flow chart of a method for processing a multimodal neural biosignal according to an embodiment of the present application. As shown in FIG3 , the process includes the following steps:
S21，获取待处理的多模态神经生物信号，对多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号。详细说明参见上述实施例对应的相关描述，此处不再赘述。S21, obtaining a multimodal neural biological signal to be processed, preprocessing the multimodal neural biological signal, and obtaining a preprocessed multimodal neural biological signal. Detailed descriptions refer to the relevant descriptions of the above embodiments, which will not be repeated here.
S22，将预处理后的多模态神经生物信号输入至深度学习模型中，基于深度学习模型提取多模态神经生物信号中每种模态神经生物信号的深度特征。S22, input the preprocessed multimodal neural biological signal into the deep learning model, and extract the deep features of each modality of the multimodal neural biological signal based on the deep learning model.
具体地，深度学习模型中预先构建有多个深度学习子模型，不同的深度学习子模型用于不同类型的神经生物信号，各个深度学习子模型均是基于相应的神经生物信号样本预先训练得到的。相应地，上述步骤S22可以包括：Specifically, a plurality of deep learning sub-models are pre-built in the deep learning model, and different deep learning sub-models are used for different types of neural biological signals, and each deep learning sub-model is pre-trained based on the corresponding neural biological signal samples. Accordingly, the above step S22 may include:
S221，确定多模态神经生物信号中的类型。S221, determining types in multimodal neural biosignals.
如上文所述，多模态神经生物信号中包括有多种模态的神经生物信号，不同模态的神经生物信号对应的特征信息不同。在将多模态神经生物信号输入至深度学习网络时，通过多模态神经生物信号的特征信息区分出不同类型的神经生物信号。As mentioned above, multimodal neural biological signals include neural biological signals of multiple modes, and neural biological signals of different modes correspond to different feature information. When multimodal neural biological signals are input into a deep learning network, different types of neural biological signals are distinguished by the feature information of the multimodal neural biological signals.
S222，确定与各个类型的神经生物信号所对应的各个深度学习子模型。S222, determining each deep learning sub-model corresponding to each type of neural biological signal.
深度学习子模型与神经生物信号的类型相对应，根据该对应关系即可确定出与各个类型的神经生物信号相对应的深度学习子模型。例如，如图4所示，多模态神经生物信号中包括有A信号、B信号和C信号，且A信号、B信号和C信号属于不同类型的信号，深度学习模型包括有子模型a、子模型b以及子模型c。其中，A信号对应于子模型a；B信号对应于子模型b；C信号对应于子模型c。The deep learning sub-model corresponds to the type of neural biological signal, and the deep learning sub-model corresponding to each type of neural biological signal can be determined based on the corresponding relationship. For example, as shown in Figure 4, the multimodal neural biological signal includes A signal, B signal and C signal, and A signal, B signal and C signal belong to different types of signals, and the deep learning model includes sub-model a, sub-model b and sub-model c. Among them, A signal corresponds to sub-model a; B signal corresponds to sub-model b; C signal corresponds to sub-model c.
S223，分别将各个类型的神经生物信号输入至相应的深度学习子模型中，得到与各个类型的神经生物信号相对应的深度特征。S223, input each type of neural biological signal into the corresponding deep learning sub-model respectively to obtain the deep features corresponding to each type of neural biological signal.
深度特征为针对于神经生物信号所生成的高维度特征，通过高维的特征矩阵对深度特征进行表示。Deep features are high-dimensional features generated for neural biological signals, and are represented by high-dimensional feature matrices.
根据上述步骤确定出的神经生物信号的类型以及相应的深度学习子模型，可以将各个类型的神经生物信号输入到相应的深度学习子模型中。例如，将A信号输入至子模型a中，将B信号输入至子模型b中，并将C信号输入至子模型c中，从而得到相应的深度特征。According to the types of neural biological signals and the corresponding deep learning sub-models determined in the above steps, each type of neural biological signal can be input into the corresponding deep learning sub-model. For example, input signal A into sub-model a, input signal B into sub-model b, and input signal C into sub-model c, thereby obtaining the corresponding deep features.
当神经生物信号为第一类型神经生物信号时，与第一类型神经生物信号对应的第一深度学习子模型包括多个第一子模型串联组成的模型组以及第二子模型，由此实现了针对于第一深度学习子模型的多层深度结构设计。When the neural biological signal is a first type of neural biological signal, the first deep learning sub-model corresponding to the first type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a second sub-model, thereby realizing a multi-layer deep structure design for the first deep learning sub-model.
相应地，将第一类型神经生物信号输入至第一深度学习子模型中，得到与第一类型神经生物信号相对应的深度特征，包括：Accordingly, the first type of neural biological signal is input into the first deep learning sub-model to obtain the deep features corresponding to the first type of neural biological signal, including:
(1)将第一类型神经生物信号输入至第二子模型进行特征提取，得到第一特征数据。(1) Inputting the first type of neural biological signal into the second sub-model for feature extraction to obtain first feature data.
(2)将第一特征数据与预配置的位置编码进行融合，得到融合数据。(2) The first feature data is fused with the preconfigured position code to obtain fused data.
(3)将融合数据输入至模型组进行特征提取，得到第一类型神经生物信号相对应的深度特征。(3) The fused data is input into the model group for feature extraction to obtain the deep features corresponding to the first type of neural biological signal.
第二子模型用于初步提取第一类型神经生物信号所具有的神经生物特征。多个第一子模型构建的模型组用于进一步提取神经生物信号的深度特征。The second sub-model is used to preliminarily extract the neurobiological features of the first type of neurobiological signal. The model group constructed by multiple first sub-models is used to further extract the deep features of the neurobiological signal.
将通过第二子模型初步提取到的神经生物特征作为第一特征数据与位置编码进行融合，得到神经生物特征与位置编码的融合数据。此处通过在神经生物特征中设置位置编码，便于确定第一类型神经生物信号中各段信号所处的位置或所采集的位置。以EEG信号为例，可以通过一定的编码规则对各个采集电极的位置进行编码处理，并将其融合至初步提取出的神经生物特征中，便于后续确定出各个采集电极位置处的深度特征。The neurobiological features initially extracted by the second sub-model are used as the first feature data and fused with the position code to obtain fused data of the neurobiological features and the position code. Here, by setting the position code in the neurobiological features, it is convenient to determine the position of each signal segment in the first type of neurobiological signal or the collected position. Taking the EEG signal as an example, the position of each collection electrode can be encoded and processed according to certain coding rules, and fused into the initially extracted neurobiological features, so as to facilitate the subsequent determination of the depth features at the positions of each collection electrode.
继而将上述得到的融合数据输入至模型组中进行高维度的特征提取，以从中提取出能够表征脑电信号时空特征以及长程依赖关系的特征。将通过模型组输出的特征确定为第一类型神经生物信号的深度特征。Then, the fused data obtained above is input into the model group for high-dimensional feature extraction, so as to extract features that can characterize the spatiotemporal features and long-range dependencies of the EEG signal. The features output by the model group are determined as the deep features of the first type of neural biological signal.
此处对第一子模型和第二子模型不作限定，只要能够实现对于深度特征的提取即可，本领域技术人员可以根据实际需求确定。There is no limitation on the first sub-model and the second sub-model here, as long as the extraction of deep features can be achieved, technical personnel in this field can determine it according to actual needs.
在一个具体的实施方式中，第一类型神经生物信号为脑电信号，第一子模型包括Transformer模型；第二子模型包括卷积神经网络模型。如图5所示，卷积神经网络包括卷积层以及平均池化层。Transformer 模型包括注意力模块和前馈网络模块。其中，注意力模块包括多头注意力层以及第一归一化层，前馈网络模块包括全连接前馈层以及第二归一化层；多头注意力层与第一归一化层连接；全连接前馈层与第二归一化层连接；第一归一化层与全连接前馈层连接；注意力模块和前馈网络模块具有对应的快捷连接。In a specific embodiment, the first type of neural biological signal is an electroencephalogram signal, the first sub-model includes a Transformer model, and the second sub-model includes a convolutional neural network model. As shown in Figure 5, the convolutional neural network includes a convolutional layer and an average pooling layer. The Transformer model includes an attention module and a feedforward network module. Among them, the attention module includes a multi-head attention layer and a first normalization layer, and the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer; the attention module and the feedforward network module have corresponding shortcut connections.
基于多头注意机制的Transformer模型具有广泛的适用性、良好的可解释性及善于扑捉序列信号中的长程依赖关系等优势。为了更加准确地挖掘和提取脑电信号的时空特征及长程依赖关系，此处可以选择将Transformer模型与卷积神经网络模型(Convolutional Neural Networks，CNN)进行融合，图5是基于Transformer模型融合CNN模型得到CNN-Trans模型的框图。The Transformer model based on the multi-head attention mechanism has the advantages of wide applicability, good interpretability, and good at capturing long-range dependencies in sequence signals. In order to more accurately mine and extract the spatiotemporal features and long-range dependencies of EEG signals, the Transformer model can be fused with the Convolutional Neural Networks (CNN) model. Figure 5 is a block diagram of the CNN-Trans model obtained by fusing the Transformer model with the CNN model.
其中，图5(a)是Transformer模型的结构图，Transformer模型包括注意力模块和前馈网络模块，注意力模块包括多头注意力层，之后紧接第一归一化层；前馈网络模块包括全连接的前馈层，之后紧接第二归一化层。注意力模块和前馈网络模块的周围采用快捷连接。该注意力模块包括多头注意力层，允许Transformer模型共同注意来自不同位置的信息，例如使用8头注意力层(即8个注意力头)。此处对注意力层的个数不作限定，本领域技术人员可以根据实际需求确定。Among them, Figure 5(a) is a structural diagram of the Transformer model. The Transformer model includes an attention module and a feedforward network module. The attention module includes a multi-head attention layer, followed by a first normalization layer; the feedforward network module includes a fully connected feedforward layer, followed by a second normalization layer. Quick connections are used around the attention module and the feedforward network module. The attention module includes a multi-head attention layer, which allows the Transformer model to jointly pay attention to information from different positions, such as using an 8-head attention layer (i.e., 8 attention heads). The number of attention layers is not limited here, and those skilled in the art can determine it according to actual needs.
图5(b)是CNN模型的结构图，CNN模型包括：卷积层和平均池化层。如图5(c)所示，将CNN模型融合位置编码后输入到多个Transformer模型串联所得到的模型组中即构建出CNN-Trans模型结构。例如，3个Transformer模型串联所得到的模型组。此处对Transformer模型的串联个数不作限定，本领域技术人员可以根据实际需求确定。FIG5(b) is a structural diagram of a CNN model, which includes a convolutional layer and an average pooling layer. As shown in FIG5(c), the CNN model fusion position encoding is input into a model group obtained by connecting multiple Transformer models in series to construct a CNN-Trans model structure. For example, a model group obtained by connecting three Transformer models in series. The number of Transformer models connected in series is not limited here, and those skilled in the art can determine it according to actual needs.
通过CNN-Trans模型即可提取脑电信号的深度特征。此处，可以采用迁移学习方法，将经过正常人体脑电样本训练过的CNN-Trans模型作为初始化模型，以有效提高CNN-Trans模型训练的效率和精度。The deep features of EEG signals can be extracted through the CNN-Trans model. Here, the transfer learning method can be used to use the CNN-Trans model trained with normal human EEG samples as the initialization model to effectively improve the efficiency and accuracy of CNN-Trans model training.
当神经生物信号为第二类型神经生物信号时，与第二类型神经生物信号对应的第二深度学习子模型包括多个第一子模型串联组成的模型组以及第三子模型，第三子模型包括卷积层、最大池化层及全连接层，由此实现了针对于第二深度学习子模型的多层深度结构设计。When the neural biological signal is a second type of neural biological signal, the second deep learning sub-model corresponding to the second type of neural biological signal includes a model group composed of multiple first sub-models connected in series and a third sub-model, and the third sub-model includes a convolutional layer, a maximum pooling layer and a fully connected layer, thereby realizing a multi-layer deep structure design for the second deep learning sub-model.
相应地，将第二类型神经生物信号输入至第二深度学习子模型中，得到与第二类型神经生物信号相对应的深度特征，包括：Accordingly, the second type of neural biological signal is input into the second deep learning sub-model to obtain the deep features corresponding to the second type of neural biological signal, including:
(1)将第二类型神经生物信号输入至第三子模型的卷积层进行多维卷积处理，得到卷积处理结果。(1) The second type of neural biological signal is input into the convolution layer of the third sub-model for multi-dimensional convolution processing to obtain the convolution processing result.
(2)将卷积处理结果输入至模型组进行特征提取，并经过最大池化层与全连接层，输出第二类型神经生物信号相对应的深度特征。(2) The convolution processing result is input into the model group for feature extraction, and after passing through the maximum pooling layer and the fully connected layer, the deep features corresponding to the second type of neural biological signal are output.
第三子模型用于对第二类型神经生物信号进行多维卷积处理，多个第一子模型串联构成的模型组用于提取第二类型神经生物信号中深度特征。The third sub-model is used to perform multi-dimensional convolution processing on the second type of neural biological signal, and the model group formed by connecting multiple first sub-models in series is used to extract deep features from the second type of neural biological signal.
通过第三子模型对第二类型神经生物特征进行多维卷积处理，以将第二类型神经生物信号转换为多维矩阵表征的信号，并将经过多维卷积处理得到的卷积处理结果输入至模型组中进行高维度的特征提取，以从中提取出能够表征功能磁共振信号的空间及功能连接的特征。将经过模型组输出的高维度特征经由最大池化层与全连接层后输出，得到第二类型神经生物信号的深度特征。The third sub-model performs multi-dimensional convolution processing on the second type of neurobiological features to convert the second type of neurobiological signals into signals represented by multi-dimensional matrices, and the convolution processing results obtained by the multi-dimensional convolution processing are input into the model group for high-dimensional feature extraction to extract the features that can characterize the spatial and functional connection of the functional magnetic resonance signal. The high-dimensional features output by the model group are output through the maximum pooling layer and the fully connected layer to obtain the deep features of the second type of neurobiological signals.
此处对第一子模型和第三子模型不作限定，只要能够实现对于深度特征的提取即可，本领域技术人员可以根据实际需求确定。There is no limitation on the first sub-model and the third sub-model here, as long as they can realize the extraction of deep features, those skilled in the art can determine them according to actual needs.
在一个具体的实施方式中，第二类型神经生物信号为功能磁共振信号，第一子模型包括Transformer模型；第三子模型包括点4D卷积网络模型。如图6所示，点4D卷积网络模型包括点4D卷积层、最大池化层及全连接层；多个Transformer模型串联组成的模型组设置在点4D卷积层与最大池化层之间；Transformer模型包括注意力模块和前馈网络模块；注意力模块包括多头注意力层以及第一归一化层；前馈网络模块包括全连接前馈层以及第二归一化层；多头注意力层与第一归一化层连接；全连接前馈层与第二归一化层连接；第一归一化层与全连接前馈层连接；注意力模块和前馈网络模块具有对应的快捷连接。In a specific embodiment, the second type of neural biological signal is a functional magnetic resonance signal, the first sub-model includes a Transformer model; and the third sub-model includes a point 4D convolutional network model. As shown in FIG6 , the point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer, and a fully connected layer; a model group consisting of multiple Transformer models connected in series is set between the point 4D convolutional layer and the maximum pooling layer; the Transformer model includes an attention module and a feedforward network module; the attention module includes a multi-head attention layer and a first normalization layer; the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer; the attention module and the feedforward network module have corresponding shortcut connections.
为了更加准确地挖掘和提取功能磁共振信号的空间及功能连接等特征，此处可以选择将Transformer模型与点4D卷积网络模型进行融合，图6是基于Transformer模型融合点4D卷积网络模型得到4D-Trans模型的框图。In order to more accurately mine and extract the spatial and functional connectivity features of fMRI signals, we can choose to fuse the Transformer model with the point 4D convolutional network model. Figure 6 is a block diagram of the 4D-Trans model obtained by fusing the point 4D convolutional network model with the Transformer model.
其中，Transformer模型包括注意力模块和前馈网络模块，注意力模块包括多头注意力层，之后紧接 第一归一化层；前馈网络模块包括全连接的前馈层，之后紧接第二归一化层。注意力模块和前馈网络模块的周围采用快捷连接，具体参见图5所示。通过Transformer模型根据需要将相应的二维操作改为三维/四维操作。The Transformer model includes an attention module and a feedforward network module. The attention module includes a multi-head attention layer followed by a first normalization layer. The feedforward network module includes a fully connected feedforward layer followed by a second normalization layer. Quick connections are used around the attention module and the feedforward network module, as shown in Figure 5. The corresponding two-dimensional operations are converted to three-dimensional/four-dimensional operations as needed through the Transformer model.
点4D卷积网络模型包括点4D卷积层、最大池化层及全连接层。如图6所示，将点4D卷积网络模型的点4D卷积层的输出结果作为多个Transformer模型串联所得到的模型组的输入，并将模型组的输出结果输入至最大池化层及全连接层中即可构建出4D-Trans模型结构。例如，3个Transformer模型串联所得到的模型组。此处对Transformer模型的串联个数不作限定，本领域技术人员可以根据实际需求确定。The point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer and a fully connected layer. As shown in Figure 6, the output result of the point 4D convolutional layer of the point 4D convolutional network model is used as the input of the model group obtained by connecting multiple Transformer models in series, and the output result of the model group is input into the maximum pooling layer and the fully connected layer to construct a 4D-Trans model structure. For example, a model group is obtained by connecting three Transformer models in series. There is no limitation on the number of Transformer models connected in series here, and those skilled in the art can determine it according to actual needs.
通过这个4D-Trans模型即可提取功能磁共振信号的深度特征。此处，可以采用迁移学习方法，将经过正常人体fMRI样本训练过的4D-Trans模型作为初始化模型，以有效提高4D-Trans模型训练的效率和精度。The deep features of functional magnetic resonance signals can be extracted through this 4D-Trans model. Here, a transfer learning method can be used to use the 4D-Trans model trained with normal human fMRI samples as an initialization model to effectively improve the efficiency and accuracy of 4D-Trans model training.
S23，将多种深度特征输入至特征融合层进行特征融合，得到目标融合特征。详细说明参见上述实施例对应的相关描述，此处不再赘述。S23, inputting multiple depth features into the feature fusion layer for feature fusion to obtain target fusion features. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.
S24，将目标融合特征经由全连接层后输入至回归层，利用回归层对目标融合特征进行体征预测，生成生物体征预测结果。详细说明参见上述实施例对应的相关描述，此处不再赘述。S24, input the target fusion feature into the regression layer after passing through the fully connected layer, and use the regression layer to perform physical sign prediction on the target fusion feature to generate a biological sign prediction result. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.
本实施例提供的多模态神经生物信号的处理方法，通过将不同类型的神经生物信号输入至相应的深度学习子模型以进行深度特征提取，便于定向提取各个类型的神经生物信号的深度特征，提高了深度特征的提取精度。The multimodal neural biological signal processing method provided in this embodiment facilitates the directional extraction of deep features of each type of neural biological signal by inputting different types of neural biological signals into corresponding deep learning sub-models for deep feature extraction, thereby improving the extraction accuracy of deep features.
在本实施例中提供了一种多模态神经生物信号的处理方法，可用于深度学习网络中，该深度学习网络包括预先构建的深度学习模型、深度回归模型，深度回归模型包括特征融合层、全连接层和回归层。In this embodiment, a method for processing multimodal neural biological signals is provided, which can be used in a deep learning network. The deep learning network includes a pre-built deep learning model and a deep regression model. The deep regression model includes a feature fusion layer, a fully connected layer and a regression layer.
图7是根据本申请实施例的多模态神经生物信号的处理方法的流程图，如图7所示，该流程包括如下步骤：FIG. 7 is a flow chart of a method for processing a multimodal neural biosignal according to an embodiment of the present application. As shown in FIG. 7 , the process includes the following steps:
S31，获取待处理的多模态神经生物信号，对多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号。详细说明参见上述实施例对应的相关描述，此处不再赘述。S31, obtaining a multimodal neural biological signal to be processed, preprocessing the multimodal neural biological signal, and obtaining a preprocessed multimodal neural biological signal. Detailed descriptions refer to the corresponding descriptions of the above embodiments, which will not be repeated here.
S32，将预处理后的多模态神经生物信号输入至深度学习模型中，基于深度学习模型提取多模态神经生物信号中每种模态神经生物信号的深度特征。详细说明参见上述实施例对应的相关描述，此处不再赘述。S32, input the preprocessed multimodal neural biological signal into the deep learning model, and extract the deep features of each modality neural biological signal in the multimodal neural biological signal based on the deep learning model. For detailed description, please refer to the relevant description corresponding to the above embodiment, which will not be repeated here.
S33，将多种深度特征输入至特征融合层进行特征融合，得到目标融合特征。详细说明参见上述实施例对应的相关描述，此处不再赘述。S33, inputting multiple depth features into the feature fusion layer for feature fusion to obtain target fusion features. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.
S34，获取目标融合特征对应的数据类型。S34, obtaining the data type corresponding to the target fusion feature.
数据类型包括离散型数据和连续型数据。回归层可以进行离散值的分类预测，也可以进行连续值的回归预测。在将目标融合特征输入至回归层进行体征预测时，可以对目标融合特征的值特征进行分析，以确定当前所得到的目标融合特征的数据类型。The data types include discrete data and continuous data. The regression layer can perform classification prediction of discrete values or regression prediction of continuous values. When the target fusion feature is input into the regression layer for sign prediction, the value feature of the target fusion feature can be analyzed to determine the data type of the current target fusion feature.
S35，构建与数据类型对应的损失函数。S35, construct a loss function corresponding to the data type.
对于不同的数据类型，可以构建不同的损失函数。Different loss functions can be constructed for different data types.
具体地，对于离散类型的数据，可以采用交叉熵(Cross-Entropy)作为损失函数(Loss)对回归层的深度回归模型进行分类训练，并在最后一层全连接层之后添加softmax激活函数，以对输出值进行归一化操作。Specifically, for discrete data, cross-entropy can be used as the loss function to classify the deep regression model of the regression layer, and a softmax activation function can be added after the last fully connected layer to normalize the output value.
具体地，当数据类型为连续类型时，可采用数据误差参数来构建损失函数(Loss)对深度回归模型进行优化训练。其中，数据误差参数包括平均绝对误差、均方根误差以及中位绝对误差，构建的损失函数为：Specifically, when the data type is continuous, the data error parameters can be used to construct the loss function (Loss) to optimize the training of the deep regression model. Among them, the data error parameters include the mean absolute error, the root mean square error, and the median absolute error. The constructed loss function is:
Loss＝α*RMSE+β*MAE+μ*MedAE+λ*||W||；Loss = α*RMSE+β*MAE+μ*MedAE+λ*||W||;
其中，Loss表示损失函数；RMSE表示均方根误差；MAE表示平均绝对误差；MedAE表示中位绝对误差；||W||表示正则化项；α、β、μ、λ表示网络参数。Among them, Loss represents the loss function; RMSE represents the root mean square error; MAE represents the mean absolute error; MedAE represents the median absolute error; ||W|| represents the regularization term; α, β, μ, and λ represent network parameters.
由于RMSE、MAE、MedAE是3个不同的评价指标，此处综合考虑这3个评价指标，并将这3个指标融合到Loss函数中以提高回归精度。||W||是与网络参数相关的正则化项，通过添加正则化项到Loss函 数中能够在一定程度上减少过拟合程度。Since RMSE, MAE, and MedAE are three different evaluation indicators, these three evaluation indicators are comprehensively considered here and integrated into the Loss function to improve the regression accuracy. ||W|| is a regularization term related to the network parameters. By adding the regularization term to the Loss function, the degree of overfitting can be reduced to a certain extent.
α、β、μ、λ需要经过网络训练确定。具体的，可以采用误差反向传播算法计算损失函数对网络参数的梯度，并配合优化方法更新网络参数以降低损失函数，直至损失函数降低至最低值，网络收敛，将此时网络参数确定出的最优的网络参数。α, β, μ, and λ need to be determined through network training. Specifically, the error back propagation algorithm can be used to calculate the gradient of the loss function to the network parameters, and the network parameters can be updated with the optimization method to reduce the loss function until the loss function is reduced to the minimum value and the network converges. The network parameters at this time are determined to be the optimal network parameters.
需要说明的是，在数据类型为连续类型的回归预测中，最后一层全连接层之后并不会添加激活函数，或者只添加线性激活函数，在其他网络层可以根据需要添加线性或非线性激活函数。对于回归层输出的连续预测值同样需要归一化到[0-1]之间，以使深度学习模型更容易训练和收敛，保证该模型能够获得更好的预测结果。It should be noted that in the regression prediction of continuous data type, no activation function is added after the last fully connected layer, or only a linear activation function is added. Linear or nonlinear activation functions can be added in other network layers as needed. The continuous prediction value output by the regression layer also needs to be normalized to [0-1] to make the deep learning model easier to train and converge, ensuring that the model can obtain better prediction results.
S36，利用损失函数对深度回归网络的参数进行优化处理，其中，回归层部署在该深度回归网络中。S36, optimizing the parameters of the deep regression network using the loss function, wherein the regression layer is deployed in the deep regression network.
在得到损失函数后，可以通过损失函数对深度回归网络的所有参数进行优化处理，直至损失函数降低至最低值，网络收敛，将此时确定出的各个网络参数确定为最优的网络参数，并按照最优的网络参数部署深度回归网络模型。After obtaining the loss function, all parameters of the deep regression network can be optimized through the loss function until the loss function is reduced to the minimum value and the network converges. The various network parameters determined at this time are determined as the optimal network parameters, and the deep regression network model is deployed according to the optimal network parameters.
S37，将目标融合特征经由全连接层后输入至回归层，利用回归层对目标融合特征进行体征预测，生成生物体征预测结果。详细说明参见上述实施例对应的相关描述，此处不再赘述。S37, input the target fusion feature into the regression layer after passing through the fully connected layer, and use the regression layer to perform physical sign prediction on the target fusion feature to generate a biological sign prediction result. For detailed description, please refer to the relevant description of the above embodiment, which will not be repeated here.
S38，获取生物体征预测结果所处的数值范围。S38, obtaining the numerical range of the biological sign prediction result.
生物体征预测结果包括离散值预测结果和连续值预测结果。离散值预测结果中针对于离散值的最小值和最大值构成离散值预测结果对应的数值范围，连续值预测结果为针对于连续数据值所构成的数值范围。The biological sign prediction results include discrete value prediction results and continuous value prediction results. In the discrete value prediction results, the minimum and maximum values of the discrete values constitute the numerical range corresponding to the discrete value prediction results, and the continuous value prediction results are the numerical ranges constituted by the continuous data values.
S39，对数值范围进行量化处理，将数值范围划分为若干个区间，得到对应于若干个区间的预测级别。S39, quantizing the numerical range, dividing the numerical range into a plurality of intervals, and obtaining prediction levels corresponding to the plurality of intervals.
对于离散值而言，此处的生物体征预测结果相当于分类，此处可以设置不同的预测级别，以对应于离散值的分类结果。例如，0-完全无效、1-轻微有效、2-有效、3-非常有效。For discrete values, the biological sign prediction results here are equivalent to classification, and different prediction levels can be set here to correspond to the classification results of discrete values. For example, 0-completely ineffective, 1-slightly effective, 2-effective, 3-very effective.
对于连续值而言，可以对数值范围进行量化处理，将数值范围划分为多个区间进行分类训练预测，以实现不同区间对应的预测级别的确定。例如，数值范围为[-5，15]，此时可以将数值范围划分为4个区间，并为每个区间设定相应的预测级别。具体地，将区间[-5，0]设定为0-完全无效，将区间[0，5]设定为1-轻微有效，将区间[5，10]设定为2-有效，将区间[10，15]设定为3-非常有效。最后利用准确率(Accuracy)、精确率(Precision)、F1值及AUC值等对预测结果进行综合评估。For continuous values, the numerical range can be quantified and divided into multiple intervals for classification training and prediction to determine the prediction level corresponding to different intervals. For example, the numerical range is [-5, 15]. At this time, the numerical range can be divided into 4 intervals, and the corresponding prediction level is set for each interval. Specifically, the interval [-5, 0] is set to 0-completely invalid, the interval [0, 5] is set to 1-slightly effective, the interval [5, 10] is set to 2-effective, and the interval [10, 15] is set to 3-very effective. Finally, the prediction results are comprehensively evaluated using accuracy, precision, F1 value and AUC value.
具体地，还可以通过计算平均绝对误差(MAE)、平均绝对百分比误差(MAPE)、均方误差(MSE)、均方误差根(RMSE)、均方误差对数(MSLE)、中位绝对误差(MedAE)及可决系数(r2score)等指标对预测结果进行综合评估。Specifically, the prediction results can be comprehensively evaluated by calculating indicators such as mean absolute error (MAE), mean absolute percentage error (MAPE), mean square error (MSE), root mean square error (RMSE), logarithm of mean square error (MSLE), median absolute error (MedAE) and coefficient of determination (r2score).
其中，MAE、MAPE、MSE、RMSE、MSLE、MedAE等指标用于评估真实值和预测值之间的差异，值越小越好。Among them, indicators such as MAE, MAPE, MSE, RMSE, MSLE, and MedAE are used to evaluate the difference between the true value and the predicted value. The smaller the value, the better.
其中，r2score是用于度量拟合优度的统计量，r2score的最大值为1，r2score的值越接近1，说明深度回归模型对观测值的拟合程度越好；反之，r2score的值越小，说明深度回归模型对观测值的拟合程度越差。Among them, r2score is a statistic used to measure the goodness of fit. The maximum value of r2score is 1. The closer the value of r2score is to 1, the better the deep regression model fits the observed value; conversely, the smaller the value of r2score is, the worse the deep regression model fits the observed value.
S310，基于预设的可解释人工智能方法从生物体征预测结果中提取生物特征。S310, extracting biological features from the biological sign prediction results based on a preset explainable artificial intelligence method.
由于基于神经生物信号异常产生的精神疾病具有高度的神经生物学异质性，常规的组分析方法难以发现能够表征精神疾病的生物特征。可解释人工智能方法为基于深度学***分析。Since mental illnesses based on abnormal neurobiological signals have a high degree of neurobiological heterogeneity, conventional group analysis methods have difficulty in discovering biological features that can characterize mental illnesses. Explainable artificial intelligence methods are an analysis method based on deep learning networks that can achieve individual-level analysis of organisms.
此处通过深度学***分析，即可找到针对于精神疾病药效反应的个体水平的生物特征。Here, the interpretable artificial intelligence method (XAI) of the deep learning network is used to analyze the biological characteristics of the drug efficacy of mental illness from the biological sign prediction results. Specifically, the interpretable artificial intelligence method (XAI) is applied to the deep regression model, and the individual level analysis of the biological sign prediction results output by the deep regression model is performed to find the individual level biological characteristics of the drug efficacy response of mental illness.
可解释人工智能方法(XAI)包括局部替代可视化方法、遮挡分析法、梯度可视化方法及分层相关性传播方法等，此处对选用的可解释人工智能方法不作限定，本领域技术人员可以根据实际需求确定。Explainable artificial intelligence methods (XAI) include local substitution visualization methods, occlusion analysis methods, gradient visualization methods, and hierarchical correlation propagation methods, etc. There is no limitation on the selected explainable artificial intelligence methods here, and technicians in this field can determine them according to actual needs.
例如，采用分层相关性传播方法(LRP)提取生物特征，具体的实现流程包括：将经过处理的神经生 物信号输入到深度回归模型中可以得到回归层的输出结果(即当前的生物体征预测结果)，基于LRP的相关性准则将回归层的输出结果的相关性从深度回归模型的输出层反向传播到输入层，这样就得到了输入数据的所有特征(数据点)的LRP值，LRP值越大说明该特征(数据点)在预测/决策过程中给予的贡献越大。基于上述流程可以生成针对于每个受试者的LRP热力图，通过LRP热力图就可以提取个体水平的生物特征。For example, the hierarchical relevance propagation method (LRP) is used to extract biological features. The specific implementation process includes: the processed neurobiological signal is input into the deep regression model to obtain the output result of the regression layer (i.e., the current biological sign prediction result), and the correlation of the output result of the regression layer is back-propagated from the output layer of the deep regression model to the input layer based on the LRP correlation criterion, so that the LRP value of all features (data points) of the input data is obtained. The larger the LRP value, the greater the contribution of the feature (data point) in the prediction/decision-making process. Based on the above process, an LRP heat map can be generated for each subject, and the biological features at the individual level can be extracted through the LRP heat map.
S311，对生物特征进行复制分析和收敛性分析，从生物特征中确定出个体生物标记。S311, performing replication analysis and convergence analysis on the biometrics to determine individual biomarkers from the biometrics.
对上述生物特征进行复制分析和收敛性分析，如图8所示，通过复制分析结果和收敛性分析结果对生物特征的可移植性和稳定性进行评估，以确定针对于药效预测的生物特征是否能够作为潜在的生物标记，同时能够通过使用其他非药物治疗的结果对药效预测结果及生物特征进行交叉验证，进一步验证了药效预测结果及生物特征的神经生物学意义和临床意义。The above-mentioned biosignatures were subjected to replication analysis and convergence analysis, as shown in FIG8 . The portability and stability of the biosignatures were evaluated through the replication analysis results and the convergence analysis results to determine whether the biosignatures for drug efficacy prediction can be used as potential biomarkers. At the same time, the drug efficacy prediction results and biosignatures can be cross-validated by using the results of other non-drug treatments, further verifying the neurobiological significance and clinical significance of the drug efficacy prediction results and biosignatures.
其中，复制分析是通过将生物体征预测结果和药效反应的生物特征应用于本地数据库，通过数据库中的跨数据集的复制分析对药效预测及生物特征的可移植性和稳定性进行评估，以确定生物特征是否可以作为生物标记。例如，采用上述所述LRP方法可以得到LRP热力图，LRP值越大的生物特征对于生物体征预测的贡献越大，将贡献大的生物特征作为潜在的生物标记。而贡献大小可以采用多种深度学习可视化分析方法计算得到的综合贡献值进行综合评估，当某个生物特征对应的综合贡献值大于设定阈值时，就可以将该生物特征作为潜在的生物标记。Among them, replication analysis is to apply the biological sign prediction results and the biological characteristics of drug efficacy response to the local database, and evaluate the portability and stability of drug efficacy prediction and biological characteristics through replication analysis across data sets in the database to determine whether the biological characteristics can be used as biomarkers. For example, the LRP heat map can be obtained by using the above-mentioned LRP method. The biological characteristics with larger LRP values have greater contributions to biological sign prediction, and biological characteristics with large contributions are used as potential biomarkers. The contribution size can be comprehensively evaluated by using the comprehensive contribution value calculated by a variety of deep learning visualization analysis methods. When the comprehensive contribution value corresponding to a certain biological characteristic is greater than the set threshold, the biological characteristic can be used as a potential biomarker.
其中，收敛性分析主要基于精神疾病药物服用前、后的多模态神经生物信号来预测精神疾病药物的疗效。但是实际临床中，受试者不仅会采用药物治疗，还会采用非药物治疗，例如经颅磁刺激TMS等物理治疗或心理治疗等，由此同一个受试者则会存在不同治疗方式下的治疗数据。对非药物治疗数据的收敛性分析可以进一步验证生物特征的神经生物学意义及临床意义。Among them, convergence analysis is mainly based on multimodal neurobiological signals before and after taking psychiatric drugs to predict the efficacy of psychiatric drugs. However, in actual clinical practice, subjects will not only use drug treatment, but also non-drug treatment, such as physical therapy or psychological therapy such as transcranial magnetic stimulation TMS, so the same subject will have treatment data under different treatment methods. Convergence analysis of non-drug treatment data can further verify the neurobiological significance and clinical significance of biological characteristics.
具体地，收敛性分析包括：将预测的药物疗效不好和实际的非药物治疗效果不好的数据进行对比分析，找到这些数据的共同点；将预测的药物疗效不好和实际的非药物治疗效果好的数据进行对比分析，找到数据的差异性；将预测的药物疗效好和实际的非药物治疗效果不好的数据进行对比分析，找到数据的差异性；将预测的药物疗效好和实际的非药物治疗效果好的数据进行对比分析，找到这些数据的共同点。对于不同治疗方式的数据通过以上的交叉对比分析确定的共同点和差异性更具有生物意义和临床意义。Specifically, convergence analysis includes: comparing and analyzing the data of poor drug efficacy predicted and poor non-drug treatment effect actually, and finding the common points of these data; comparing and analyzing the data of poor drug efficacy predicted and good non-drug treatment effect actually, and finding the difference of data; comparing and analyzing the data of good drug efficacy predicted and poor non-drug treatment effect actually, and finding the difference of data; comparing and analyzing the data of good drug efficacy predicted and good non-drug treatment effect actually, and finding the common points of these data. The common points and differences determined by the above cross-comparison analysis of data of different treatment methods are more biologically meaningful and clinically meaningful.
作为一个可选的实施方式，如图9所示，通过深度学***的生物标记，为临床医生在治疗精神疾病提供潜在的有力辅助。临床医生在实际治疗中可以将其实时采集到的临床数据通过深度学习网络和可解释人工智能方法确定出的生物标记进行复制分析，同时结合实际诊治过程中的结果也可以进一步验证、优化和修正深度学习网络中的深度学习模型和深度回归模型，实现交叉验证。As an optional implementation, as shown in FIG9 , individual-level biomarkers mined through deep learning networks and interpretable artificial intelligence methods provide potential powerful assistance to clinicians in the treatment of mental illness. In actual treatment, clinicians can replicate and analyze the clinical data collected in real time through deep learning networks and biomarkers determined by interpretable artificial intelligence methods. At the same time, combined with the results of actual diagnosis and treatment, they can further verify, optimize and correct the deep learning model and deep regression model in the deep learning network to achieve cross-validation.
本实施例提供的多模态神经生物信号的处理方法，通过构建损失函数，便于将目标融合特征学习损失与回归损失进行联合学习优化，以提高回归层的回归性能，保证回归层的分析精度。通过对数值范围进行量化处理以将其划分为若干个区间，并为不同的区间设定不同的预测级别，能够辅助临床医生进行生物表征的有效评估。通过可解释人工智能应用于生物体征预测结果进行生物特征的提取，并通过复制分析和收敛性分析以确定出个体化生物标记，对于生物特征的可移植性和稳定性具有较大的神经生物学意义，能够为面对精神疾病的临床医生提供潜在的有力辅助。The multimodal neurobiological signal processing method provided in this embodiment, by constructing a loss function, facilitates the joint learning optimization of the target fusion feature learning loss and the regression loss to improve the regression performance of the regression layer and ensure the analysis accuracy of the regression layer. By quantifying the numerical range to divide it into several intervals, and setting different prediction levels for different intervals, it can assist clinicians in effectively evaluating biological characterizations. By applying interpretable artificial intelligence to the results of biological sign predictions to extract biological features, and by replica analysis and convergence analysis to determine individualized biomarkers, it has great neurobiological significance for the portability and stability of biological features, and can provide potential powerful assistance to clinicians facing mental illness.
基于上述多模态神经生物信号的处理方法，本实施例设计了一种深度学习***，用于精神疾病药效预测和生物特征解析。如图10所示，该深度学习***主要包括：临床数据处理模块、计算模型搭建模块、生物标记解析模块及临床应用验证模块。Based on the above-mentioned multimodal neural biological signal processing method, this embodiment designs a deep learning system for predicting the efficacy of mental illness and analyzing biological characteristics. As shown in Figure 10, the deep learning system mainly includes: a clinical data processing module, a computational model building module, a biomarker analysis module, and a clinical application verification module.
其中，临床数据处理模块主要负责临床数据的处理分析，主要包括数据整理、清洗和预处理等工作。Among them, the clinical data processing module is mainly responsible for the processing and analysis of clinical data, mainly including data sorting, cleaning and preprocessing.
其中，计算模型搭建模块主要负责药物治疗效果预测的深度学习网络的构建与训练相关工作，包括：基于深度学习网络的深度学习模型以及有监督回归模型的设计、搭建、训练和测试等，并对生物体征预测结果进行定量评估。Among them, the computational model building module is mainly responsible for the construction and training of deep learning networks for predicting drug treatment effects, including: the design, construction, training and testing of deep learning models based on deep learning networks and supervised regression models, and quantitative evaluation of biological sign prediction results.
其中，生物标记解析模块主要负责个体水平的生物特征的挖掘与解析，此处基于可解释人工智能方法 (XAI)对训练好的深度学习网络输出的生物体征预测结果进行可视化分析，挖掘和解析出针对于药效反应的个体化生物特征，并在不同研究站点的数据集和不同类型的数据集上对模型和生物特征进行复制分析及收敛性验证。Among them, the biomarker analysis module is mainly responsible for the mining and analysis of biological features at the individual level. Here, based on the explainable artificial intelligence method (XAI), the biological sign prediction results output by the trained deep learning network are visualized and analyzed to mine and parse the individualized biological features for drug efficacy response, and the model and biological features are replicated and analyzed and the convergence is verified on the data sets of different research sites and different types of data sets.
其中，临床应用验证模块主要负责通过个体化生物特征以指导和辅助精神科医生的临床治疗，并对模型和生物标记进行优化及临床验证，以揭示精神疾病药物治疗的反应表型，分离精神疾病药物和安慰剂反应，促进神经生物学对精神疾病药物疗效的理解，并为潜在治疗选择提供初步证据。Among them, the clinical application verification module is mainly responsible for guiding and assisting psychiatrists' clinical treatment through individualized biological characteristics, and optimizing and clinically verifying models and biomarkers to reveal the response phenotype of psychiatric drug treatment, separate psychiatric drug and placebo responses, promote neurobiological understanding of the efficacy of psychiatric drugs, and provide preliminary evidence for potential treatment options.
需要说明的是，上述深度学习***的临床数据处理模块、计算模型搭建模块、生物标记解析模块和临床应用验证模块可集成到一个大型的医疗辅助治疗智能***中，也可各模块单独实现。上述深度学习***的临床数据集和药物疗效预测深度网络及个体化生物特征可存储于专有服务器，通过本地或云服务方式供其他模块访问，也可与其他模块整体集成于同一个***中。It should be noted that the clinical data processing module, computational model building module, biomarker analysis module and clinical application verification module of the above-mentioned deep learning system can be integrated into a large-scale medical auxiliary treatment intelligent system, or each module can be implemented separately. The clinical data set, drug efficacy prediction deep network and individualized biological characteristics of the above-mentioned deep learning system can be stored in a dedicated server and accessed by other modules through local or cloud services, or integrated with other modules into the same system as a whole.
本实施例提供的深度学***的自动化分析，从而基于深度学习网络构建的模型可以有效地辅助临床治疗。同时，基于可解释人工智能的个体化解析方法在挖掘高度异质的精神疾病的生物特征时更精准，可为临床治疗提供可推广、可复制的生物标记，能够辅助临床医生制定相应的干预措施，为精神疾病的临床治疗提供了有效支持。The deep learning system provided in this embodiment is more predictive of the drug efficacy response of mental illness through the deep learning network constructed by objective biological information and big data mining, and deep learning can be applied to automated analysis at the individual level, so that the model constructed based on the deep learning network can effectively assist clinical treatment. At the same time, the individualized analysis method based on explainable artificial intelligence is more accurate in mining the biological characteristics of highly heterogeneous mental illnesses, and can provide scalable and replicable biomarkers for clinical treatment, which can assist clinicians in formulating corresponding intervention measures and provide effective support for the clinical treatment of mental illness.
在本实施例中还提供了一种多模态神经生物信号的处理装置，该装置用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“模块”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。In this embodiment, a multimodal neural bio-signal processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" can implement a combination of software and/or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.
本实施例提供一种多模态神经生物信号的处理装置，该装置可用于深度学习网络中，该深度学习网络包括预先构建的深度学习模型、深度回归模型，深度回归模型包括特征融合层、全连接层和回归层。如图11所示，该多模态神经生物信号的处理装置，包括：This embodiment provides a multimodal neural biological signal processing device, which can be used in a deep learning network, the deep learning network includes a pre-built deep learning model and a deep regression model, the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer. As shown in Figure 11, the multimodal neural biological signal processing device includes:
获取模块41，用于获取待处理的多模态神经生物信号，对多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号。The acquisition module 41 is used to acquire the multimodal neural biological signal to be processed, preprocess the multimodal neural biological signal, and obtain the preprocessed multimodal neural biological signal.
特征提取模块42，用于将预处理后的多模态神经生物信号输入至深度学习模型中，基于深度学习模型提取多模态神经生物信号中每种模态神经生物信号的深度特征。The feature extraction module 42 is used to input the preprocessed multimodal neural biological signal into the deep learning model, and extract the deep features of each modality of the multimodal neural biological signal based on the deep learning model.
特征融合模块43，用于将多种深度特征输入至特征融合层进行特征融合，得到目标融合特征。The feature fusion module 43 is used to input multiple depth features into the feature fusion layer for feature fusion to obtain target fusion features.
预测模块44，用于将目标融合特征经由全连接层后输入至回归层，利用回归层对目标融合特征进行体征预测，生成生物体征预测结果。The prediction module 44 is used to input the target fusion features into the regression layer after passing through the fully connected layer, and use the regression layer to perform physical sign prediction on the target fusion features to generate biological sign prediction results.
可选地，深度学习模型中预先构建有多个深度学习子模型，上述特征提取模块42可以包括：Optionally, a plurality of deep learning sub-models are pre-constructed in the deep learning model, and the feature extraction module 42 may include:
类型确定子模块，用于确定多模态神经生物信号中的类型。The type determination submodule is used to determine the type in the multimodal neural biological signal.
子模型确定子模块，用于确定与各个类型的神经生物信号所对应的各个深度学习子模型。The sub-model determination sub-module is used to determine each deep learning sub-model corresponding to each type of neural biological signal.
特征提取子模块，用于分别将各个类型的神经生物信号输入至相应的深度学习子模型中，得到与各个类型的神经生物信号相对应的深度特征。The feature extraction submodule is used to input each type of neural biological signal into the corresponding deep learning submodel to obtain the deep features corresponding to each type of neural biological signal.
可选地，当神经生物信号为第一类型神经生物信号时，与第一类型神经生物信号对应的第一深度学习子模型包括多个第一子模型串联组成的模型组以及第二子模型。Optionally, when the neural bio-signal is a first type of neural bio-signal, the first deep learning sub-model corresponding to the first type of neural bio-signal includes a model group consisting of multiple first sub-models connected in series and a second sub-model.
相应地，上述特征提取子模块具体可以用于：将第一类型神经生物信号输入至第二子模型进行特征提取，得到第一特征数据；将第一特征数据与预配置的位置编码进行融合，得到融合数据；将融合数据输入至模型组进行特征提取，得到第一类型神经生物信号相对应的深度特征。Correspondingly, the above-mentioned feature extraction submodule can be specifically used for: inputting the first type of neural biological signal into the second sub-model for feature extraction to obtain first feature data; fusing the first feature data with the preconfigured position code to obtain fused data; inputting the fused data into the model group for feature extraction to obtain the deep features corresponding to the first type of neural biological signal.
具体地，第一类型神经生物信号为脑电信号，第一子模型包括Transformer模型；第二子模型包括卷积神经网络模型。如图5所示，卷积神经网络包括卷积层以及平均池化层。Transformer模型包括注意力模块和前馈网络模块。其中，注意力模块包括多头注意力层以及第一归一化层，前馈网络模块包括全连接前馈层以及第二归一化层；多头注意力层与第一归一化层连接；全连接前馈层与第二归一化层连接；第一归一化层与全连接前馈层连接。Specifically, the first type of neural biological signal is an electroencephalogram signal, the first sub-model includes a Transformer model, and the second sub-model includes a convolutional neural network model. As shown in Figure 5, the convolutional neural network includes a convolutional layer and an average pooling layer. The Transformer model includes an attention module and a feedforward network module. Among them, the attention module includes a multi-head attention layer and a first normalization layer, and the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer.
可选地，当神经生物信号为第二类型神经生物信号时，与第二类型神经生物信号对应的第二深度学习子模型包括多个第一子模型串联组成的模型组以及第三子模型，第三子模型包括卷积层、最大池化层及全连接层。Optionally, when the neural biological signal is a second type of neural biological signal, the second deep learning sub-model corresponding to the second type of neural biological signal includes a model group consisting of multiple first sub-models connected in series and a third sub-model, and the third sub-model includes a convolutional layer, a maximum pooling layer and a fully connected layer.
相应地，上述特征提取子模块具体可以用于：将第二类型神经生物信号输入至第三子模型的卷积层进行多维卷积处理，得到卷积处理结果；将卷积处理结果输入至模型组进行特征提取，并经过最大池化层与全连接层，输出第二类型神经生物信号相对应的深度特征。Correspondingly, the above-mentioned feature extraction submodule can be specifically used to: input the second type of neural biological signal into the convolution layer of the third sub-model for multi-dimensional convolution processing to obtain the convolution processing result; input the convolution processing result into the model group for feature extraction, and pass through the maximum pooling layer and the fully connected layer to output the deep features corresponding to the second type of neural biological signal.
具体地，第二类型神经生物信号为功能磁共振信号，第一子模型包括Transformer模型；第三子模型包括点4D卷积网络模型。如图6所示，点4D卷积网络模型包括点4D卷积层、最大池化层及全连接层；多个Transformer模型串联组成的模型组设置在点4D卷积层与最大池化层之间；Transformer模型包括注意力模块和前馈网络模块；注意力模块包括多头注意力层以及第一归一化层；前馈网络模块包括全连接前馈层以及第二归一化层；多头注意力层与第一归一化层连接；全连接前馈层与第二归一化层连接；第一归一化层与全连接前馈层连接。Specifically, the second type of neural biological signal is a functional magnetic resonance signal, the first sub-model includes a Transformer model; and the third sub-model includes a point 4D convolutional network model. As shown in FIG6 , the point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer, and a fully connected layer; a model group consisting of multiple Transformer models connected in series is set between the point 4D convolutional layer and the maximum pooling layer; the Transformer model includes an attention module and a feedforward network module; the attention module includes a multi-head attention layer and a first normalization layer; the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; and the first normalization layer is connected to the fully connected feedforward layer.
可选地，上述多模态神经生物信号的处理装置还可以包括：Optionally, the multimodal neural bio-signal processing device may further include:
数据类型获取模块，用于获取目标融合特征对应的数据类型。The data type acquisition module is used to obtain the data type corresponding to the target fusion feature.
损失函数构建模块，用于构建与数据类型对应的损失函数。The loss function building module is used to build the loss function corresponding to the data type.
优化模块，用于利用损失函数对深度回归网络的参数进行优化处理，该回归层部署在深度回归网络中。The optimization module is used to optimize the parameters of the deep regression network using the loss function. The regression layer is deployed in the deep regression network.
具体地，当数据类型为连续类型时，上述损失函数构建模块用于基于数据误差参数构建损失函数。其中，数据误差参数包括平均绝对误差、均方根误差以及中位绝对误差，构建的损失函数为：Specifically, when the data type is a continuous type, the above loss function construction module is used to construct a loss function based on data error parameters. The data error parameters include mean absolute error, root mean square error, and median absolute error. The constructed loss function is:
Loss＝α*RMSE+β*MAE+μ*MedAE+λ*||W||；Loss = α*RMSE+β*MAE+μ*MedAE+λ*||W||;
其中，Loss表示损失函数；RMSE表示均方根误差；MAE表示平均绝对误差；MedAE表示中位绝对误差；||W||表示正则化项；α、β、μ、λ表示网络参数。Among them, Loss represents the loss function; RMSE represents the root mean square error; MAE represents the mean absolute error; MedAE represents the median absolute error; ||W|| represents the regularization term; α, β, μ, and λ represent network parameters.
可选地，上述多模态神经生物信号的处理装置还可以包括：Optionally, the multimodal neural bio-signal processing device may further include:
数值范围获取模块，用于获取生物体征预测结果所处的数值范围。The numerical range acquisition module is used to obtain the numerical range of the biological sign prediction result.
量化模块，用于对数值范围进行量化处理，将数值范围划分为若干个区间，得到对应于若干个区间的预测级别。The quantization module is used to quantize the numerical range, divide the numerical range into several intervals, and obtain prediction levels corresponding to the several intervals.
可选地，上述多模态神经生物信号的处理装置还可以包括：Optionally, the multimodal neural bio-signal processing device may further include:
生物特征提取模块，用于基于预设的可解释人工智能方法从生物体征预测结果中提取生物特征。The biometric feature extraction module is used to extract biometric features from the biological sign prediction results based on a preset explainable artificial intelligence method.
分析模块，用于对生物特征进行复制分析和收敛性分析，从生物特征中确定出个体生物标记。The analysis module is used to perform replication analysis and convergence analysis on the biometrics and determine individual biomarkers from the biometrics.
本实施例中的多模态神经生物信号的处理装置是以功能单元的形式来呈现，这里的单元是指ASIC电路，执行一个或多个软件或固定程序的处理器和存储器，和/或其他可以提供上述功能的器件。The multimodal neural bio-signal processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and memory that executes one or more software or fixed programs, and/or other devices that can provide the above functions.
上述各模块以及各子模块的更进一步的功能描述与上述对应实施例相同，在此不再赘述。The further functional description of the above modules and sub-modules is the same as that of the above corresponding embodiments and will not be repeated here.
本实施例提供的多模态神经生物信号的处理装置，通过采用深度学习网络对多模态神经生物信号进行特征提取以及多种深度特征的融合，继而对融合得到的目标融合特征进行体征预测，得到生物特征预测结果。由此该装置能够基于客观的多模态神经生物信号对多种深度特征进行融合，以实现对于多种深度特征的有效捕捉，无需人为设计或选取特征，且深度学习网络具有多层次非线性结构，能够有效解析神经生物信号中所存在的非线性关系，从而能够从多种深度特征中解析出有效的生物体征，实现了对于神经信号的生物表征的有效预测，提高了生物表征的预测准确度。The multimodal neural biological signal processing device provided in this embodiment uses a deep learning network to extract features from multimodal neural biological signals and fuse multiple deep features, and then predicts the physical signs of the target fused features obtained by fusion to obtain biological feature prediction results. Therefore, the device can fuse multiple deep features based on objective multimodal neural biological signals to achieve effective capture of multiple deep features without artificial design or feature selection, and the deep learning network has a multi-level nonlinear structure, which can effectively analyze the nonlinear relationship in neural biological signals, so that effective biological signs can be analyzed from multiple deep features, achieving effective prediction of biological representations of neural signals and improving the accuracy of biological representation prediction.
本申请实施例还提供一种服务器，该服务器可以为服务器、电脑等，该服务器具有上述图11所示的多模态神经生物信号的处理装置。The embodiment of the present application also provides a server, which may be a server, a computer, etc. The server has a multimodal neural biological signal processing device as shown in FIG. 11 above.
请参阅图12，图12是本申请可选实施例提供的一种服务器的结构示意图，如图12所示，该服务器可以包括：至少一个处理器501，例如中央处理器(Central Processing Unit，CPU)，至少一个通信接口503，存储器504，至少一个通信总线502。其中，通信总线502用于实现这些组件之间的连接通信。其中，通信接口503可以包括显示屏(Display)、键盘(Keyboard)，可选通信接口503还可以包括标准的有线接 口、无线接口。存储器504可以是高速易挥发性随机存取存储器(Random Access Memory，RAM)，也可以是非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器。存储器504可选的还可以是至少一个位于远离前述处理器501的存储装置。其中处理器501可以结合图11所描述的装置，存储器504中存储应用程序，且处理器501调用存储器504中存储的程序代码，以用于执行上述任一方法步骤。Please refer to FIG. 12, which is a schematic diagram of the structure of a server provided by an optional embodiment of the present application. As shown in FIG. 12, the server may include: at least one processor 501, such as a central processing unit (CPU), at least one communication interface 503, a memory 504, and at least one communication bus 502. The communication bus 502 is used to realize the connection and communication between these components. The communication interface 503 may include a display screen (Display) and a keyboard (Keyboard), and the optional communication interface 503 may also include a standard wired interface and a wireless interface. The memory 504 may be a high-speed volatile random access memory (Random Access Memory, RAM) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 504 may also be at least one storage device located away from the aforementioned processor 501. The processor 501 may be combined with the device described in FIG. 11, the memory 504 stores an application program, and the processor 501 calls the program code stored in the memory 504 to perform any of the above method steps.
服务器可以运行存储于存储器504的操作***，例如Windows ServerTM、Mac OS XTM、UnixTM,LinuxTM、FreeBSDTM等，本申请对操作***的具体形式不做限制，该操作***可用于作为软件环境，支持应用程序的执行，以及用于管理服务器的软硬件资源等。The server can run an operating system stored in the memory 504, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc. The present application does not limit the specific form of the operating system. The operating system can be used as a software environment to support the execution of application programs and to manage the server's software and hardware resources.
其中，通信总线502可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。通信总线502可以分为地址总线、数据总线、控制总线等。为便于表示，图12中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。The communication bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, FIG12 only uses one thick line, but does not mean that there is only one bus or one type of bus.
其中，存储器504可以包括易失性存储器(volatile memory)，例如随机存取存储器(random-access memory，RAM)；存储器也可以包括非易失性存储器(non-volatile memory)，例如快闪存储器(flash memory)，硬盘(hard disk drive，HDD)或固态硬盘(solid-state drive，SSD)；存储器504还可以包括上述种类的存储器的组合。Among them, the memory 504 may include a volatile memory (volatile memory), such as a random-access memory (RAM); the memory may also include a non-volatile memory (non-volatile memory), such as a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); the memory 504 may also include a combination of the above-mentioned types of memory.
其中，处理器501可以是中央处理器(central processing unit，CPU)，网络处理器(network processor，NP)或者CPU和NP的组合。Among them, processor 501 can be a central processing unit (CPU), a network processor (NP) or a combination of CPU and NP.
其中，处理器501还可以进一步包括硬件芯片。上述硬件芯片可以是专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)，现场可编程逻辑门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic,GAL)或其任意组合。The processor 501 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
可选地，存储器504还用于存储程序指令。处理器501可以调用程序指令，实现如本申请上述实施例中所示的多模态神经生物信号的处理方法。Optionally, the memory 504 is also used to store program instructions. The processor 501 can call the program instructions to implement the multimodal neural bio-signal processing method shown in the above embodiments of the present application.
本申请实施例还提供了一种非暂态计算机存储介质，所述计算机存储介质存储有计算机可执行指令，该计算机可执行指令可执行上述任意方法实施例中的多模态神经生物信号的处理方法。其中，所述存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)、随机存储记忆体(Random Access Memory，RAM)、快闪存储器(Flash Memory)、硬盘(Hard Disk Drive，HDD)或固态硬盘(Solid-State Drive，SSD)等；所述存储介质还可以包括上述种类的存储器的组合。The embodiment of the present application also provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the multimodal neural biological signal processing method in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.
虽然结合附图描述了本申请的实施例，但是本领域技术人员可以在不脱离本申请的精神和范围的情况下做出各种修改和变型，这样的修改和变型均落入由所附权利要求所限定的范围之内。Although the embodiments of the present application are described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations are all within the scope defined by the appended claims.

Claims (13)

  1. 一种多模态神经生物信号的处理方法，其特征在于，用于深度学习网络中，所述深度学习网络包括预先构建的深度学习模型、深度回归模型，所述深度回归模型包括特征融合层、全连接层和回归层，所述方法包括：A method for processing multimodal neural biological signals, characterized in that it is used in a deep learning network, the deep learning network includes a pre-built deep learning model and a deep regression model, the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer, and the method includes:
    获取待处理的多模态神经生物信号，对所述多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号；Acquiring a multimodal neural biological signal to be processed, and preprocessing the multimodal neural biological signal to obtain a preprocessed multimodal neural biological signal;
    将所述预处理后的多模态神经生物信号输入至所述深度学习模型中，基于所述深度学习模型提取所述多模态神经生物信号中每种模态神经生物信号的深度特征；Inputting the preprocessed multimodal neural biological signal into the deep learning model, and extracting the deep features of each modality neural biological signal in the multimodal neural biological signal based on the deep learning model;
    将多种所述深度特征输入至所述特征融合层进行特征融合，得到目标融合特征；Inputting the multiple depth features into the feature fusion layer for feature fusion to obtain target fusion features;
    将所述目标融合特征经由所述全连接层后输入至所述回归层，利用所述回归层对所述目标融合特征进行体征预测，生成生物体征预测结果。The target fusion feature is input into the regression layer after passing through the fully connected layer, and the regression layer is used to perform physical sign prediction on the target fusion feature to generate a biological sign prediction result.
  2. 根据权利要求1所述的方法，其特征在于，所述深度学习模型中构建有多个深度学习子模型；将所述多模态神经生物信号输入至所述深度学习模型中，基于所述深度学习模型提取所述多模态神经生物信号中每种模态神经生物信号的深度特征，包括：The method according to claim 1 is characterized in that a plurality of deep learning sub-models are constructed in the deep learning model; the multimodal neural biological signal is input into the deep learning model, and the deep features of each modality neural biological signal in the multimodal neural biological signal are extracted based on the deep learning model, comprising:
    确定多模态神经生物信号中的类型；Identify patterns in multimodal neurobiological signals;
    确定与各个类型的神经生物信号所对应的各个深度学习子模型；Determine each deep learning sub-model corresponding to each type of neural biological signal;
    分别将所述各个类型的神经生物信号输入至相应的深度学习子模型中，得到与所述各个类型的神经生物信号相对应的深度特征。The various types of neural biological signals are respectively input into the corresponding deep learning sub-models to obtain deep features corresponding to the various types of neural biological signals.
  3. 根据权利要求2所述的方法，其特征在于，当所述神经生物信号为第一类型神经生物信号时，与所述第一类型神经生物信号对应的第一深度学习子模型包括多个第一子模型串联组成的模型组以及第二子模型；将所述第一类型神经生物信号输入至所述第一深度学习子模型中，得到与所述第一类型神经生物信号相对应的深度特征，包括：The method according to claim 2 is characterized in that, when the neural biological signal is a first type neural biological signal, the first deep learning sub-model corresponding to the first type neural biological signal includes a model group consisting of a plurality of first sub-models connected in series and a second sub-model; the first type neural biological signal is input into the first deep learning sub-model to obtain a deep feature corresponding to the first type neural biological signal, including:
    将所述第一类型神经生物信号输入至第二子模型进行特征提取，得到第一特征数据；Inputting the first type of neural biological signal into the second sub-model for feature extraction to obtain first feature data;
    将所述第一特征数据与预配置的位置编码进行融合，得到融合数据；Fusing the first feature data with a preconfigured position code to obtain fused data;
    将所述融合数据输入至所述模型组进行特征提取，得到所述第一类型神经生物信号相对应的深度特征。The fused data is input into the model group for feature extraction to obtain the depth features corresponding to the first type of neural biological signal.
  4. 根据权利要求3所述的方法，其特征在于，所述第一类型神经生物信号为脑电信号；所述第一子模型包括Transformer模型；所述第二子模型包括卷积神经网络模型；The method according to claim 3 is characterized in that the first type of neural biological signal is an electroencephalogram signal; the first sub-model includes a Transformer model; and the second sub-model includes a convolutional neural network model;
    所述卷积神经网络包括卷积层以及平均池化层；The convolutional neural network includes a convolution layer and an average pooling layer;
    所述Transformer模型包括注意力模块和前馈网络模块；所述注意力模块包括多头注意力层以及第一归一化层；所述前馈网络模块包括全连接前馈层以及第二归一化层；所述多头注意力层与所述第一归一化层连接；所述全连接前馈层与所述第二归一化层连接；所述第一归一化层与所述全连接前馈层连接；所述注意力模块和所述前馈网络模块具有对应的快捷连接。The Transformer model includes an attention module and a feedforward network module; the attention module includes a multi-head attention layer and a first normalization layer; the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer; the attention module and the feedforward network module have corresponding shortcut connections.
  5. 根据权利要求2所述的方法，其特征在于，当所述神经生物信号为第二类型神经生物信号时，与所述第二类型神经生物信号对应的第二深度学习子模型包括多个第一子模型串联组成的模型组以及第三子模型，所述第三子模型包括卷积层、最大池化层及全连接层；将所述第二类型神经生物信号输入至第二深度学习子模型中，得到与所述第二类型神经生物信号相对应的深度特征，包括：The method according to claim 2 is characterized in that, when the neural biological signal is a second type of neural biological signal, the second deep learning sub-model corresponding to the second type of neural biological signal includes a model group consisting of multiple first sub-models connected in series and a third sub-model, and the third sub-model includes a convolutional layer, a maximum pooling layer and a fully connected layer; the second type of neural biological signal is input into the second deep learning sub-model to obtain a deep feature corresponding to the second type of neural biological signal, including:
    将所述第二类型神经生物信号输入至所述第三子模型的卷积层进行多维卷积处理，得到卷积处理结果；Inputting the second type of neural biological signal into the convolution layer of the third sub-model to perform multi-dimensional convolution processing to obtain a convolution processing result;
    将所述卷积处理结果输入至所述模型组进行特征提取，并经过所述最大池化层与所述全连接层，输出所述第二类型神经生物信号相对应的深度特征。The convolution processing result is input into the model group for feature extraction, and passes through the maximum pooling layer and the fully connected layer to output the depth features corresponding to the second type of neural biological signal.
  6. 根据权利要求5所述的方法，其特征在于，所述第二类型神经生物信号为功能磁共振信号；所述第一子模型包括Transformer模型；所述第三子模型包括点4D卷积网络模型；The method according to claim 5, characterized in that the second type of neural biological signal is a functional magnetic resonance signal; the first sub-model includes a Transformer model; the third sub-model includes a point 4D convolutional network model;
    所述点4D卷积网络模型包括点4D卷积层、最大池化层及全连接层；The point 4D convolutional network model includes a point 4D convolutional layer, a maximum pooling layer and a fully connected layer;
    多个所述Transformer模型串联组成的模型组设置在所述点4D卷积层与最大池化层之间；A model group consisting of a plurality of the Transformer models connected in series is arranged between the point 4D convolution layer and the maximum pooling layer;
    所述Transformer模型包括注意力模块和前馈网络模块；所述注意力模块包括多头注意力层以及第一归一化层；所述前馈网络模块包括全连接前馈层以及第二归一化层；所述多头注意力层与所述第一归一化层连接；所述全连接前馈层与所述第二归一化层连接；所述第一归一化层与所述全连接前馈层连接；所述注意力模块和所述前馈网络模块具有对应的快捷连接。The Transformer model includes an attention module and a feedforward network module; the attention module includes a multi-head attention layer and a first normalization layer; the feedforward network module includes a fully connected feedforward layer and a second normalization layer; the multi-head attention layer is connected to the first normalization layer; the fully connected feedforward layer is connected to the second normalization layer; the first normalization layer is connected to the fully connected feedforward layer; the attention module and the feedforward network module have corresponding shortcut connections.
  7. 根据权利要求1所述的方法，其特征在于，所述将所述目标融合特征经由所述全连接层后输入至所述回归层，利用所述回归层对所述目标融合特征进行体征预测，生成生物体征预测结果之前，还包括：The method according to claim 1 is characterized in that, before the target fusion feature is input into the regression layer via the fully connected layer, and the target fusion feature is predicted by the regression layer, and the biological sign prediction result is generated, it also includes:
    获取所述目标融合特征对应的数据类型；Obtaining the data type corresponding to the target fusion feature;
    构建与所述数据类型对应的损失函数；Constructing a loss function corresponding to the data type;
    利用所述损失函数对深度回归网络的参数进行优化处理，所述回归层部署在所述深度回归网络中。The loss function is used to optimize the parameters of the deep regression network, and the regression layer is deployed in the deep regression network.
  8. 根据权利要求7所述的方法，其特征在于，所述构建与所述数据类型对应的损失函数，包括：The method according to claim 7, characterized in that the constructing a loss function corresponding to the data type comprises:
    当所述数据类型为连续类型时，基于数据误差参数构建损失函数；When the data type is a continuous type, constructing a loss function based on a data error parameter;
    其中，所述数据误差参数包括平均绝对误差、均方根误差以及中位绝对误差，所述损失函数为：The data error parameters include mean absolute error, root mean square error and median absolute error, and the loss function is:
    Loss＝α*RMSE+β*MAE+μ*MedAE+λ*||W||；Loss = α*RMSE+β*MAE+μ*MedAE+λ*||W||;
    其中，Loss表示损失函数；RMSE表示均方根误差；MAE表示平均绝对误差；MedAE表示中位绝对误差；||W||表示正则化项；α、β、μ、λ表示网络参数。Among them, Loss represents the loss function; RMSE represents the root mean square error; MAE represents the mean absolute error; MedAE represents the median absolute error; ||W|| represents the regularization term; α, β, μ, and λ represent network parameters.
  9. 根据权利要求1所述方法，其特征在于，还包括：The method according to claim 1, further comprising:
    获取所述生物体征预测结果所处的数值范围；Obtaining the numerical range of the biological sign prediction result;
    对所述数值范围进行量化处理，将所述数值范围划分为若干个区间，得到对应于所述若干个区间的预测级别。The numerical range is quantized and divided into a plurality of intervals to obtain prediction levels corresponding to the plurality of intervals.
  10. 根据权利要求1所述的方法，其特征在于，还包括：The method according to claim 1, further comprising:
    基于预设的可解释人工智能方法从所述生物体征预测结果中提取生物特征；Extracting biological features from the biological sign prediction results based on a preset explainable artificial intelligence method;
    对所述生物特征进行复制分析和收敛性分析，从所述生物特征中确定出个体生物标记。Replication analysis and convergence analysis are performed on the biometrics, and individual biomarkers are determined from the biometrics.
  11. 一种多模态神经生物信号的处理装置，其特征在于，用于深度学习网络中，所述深度学习网络包括预先构建的深度学习模型、深度回归模型，所述深度回归模型包括特征融合层、全连接层和回归层，所述装置包括：A multimodal neural biological signal processing device, characterized in that it is used in a deep learning network, the deep learning network includes a pre-built deep learning model and a deep regression model, the deep regression model includes a feature fusion layer, a fully connected layer and a regression layer, and the device includes:
    获取模块，用于获取待处理的多模态神经生物信号，对所述多模态神经生物信号进行预处理，得到预处理后的多模态神经生物信号；An acquisition module, used for acquiring a multimodal neural biological signal to be processed, and preprocessing the multimodal neural biological signal to obtain a preprocessed multimodal neural biological signal;
    特征提取模块，用于将所述预处理后的多模态神经生物信号输入至所述深度学习模型中，基于所述深度学习模型提取所述多模态神经生物信号中每种模态神经生物信号的深度特征；A feature extraction module, used for inputting the preprocessed multimodal neural biological signal into the deep learning model, and extracting the deep features of each modality of the multimodal neural biological signal based on the deep learning model;
    特征融合模块，用于将多种所述深度特征输入至所述特征融合层进行特征融合，得到目标融合特征；A feature fusion module, used for inputting the multiple deep features into the feature fusion layer for feature fusion to obtain target fusion features;
    预测模块，用于将所述目标融合特征经由所述全连接层后输入至所述回归层，利用所述回归层对所述目标融合特征进行体征预测，生成生物体征预测结果。The prediction module is used to input the target fusion feature into the regression layer after passing through the fully connected layer, and use the regression layer to perform physical sign prediction on the target fusion feature to generate a biological sign prediction result.
  12. 一种服务器，其特征在于，包括：A server, comprising:
    存储器和处理器，所述存储器和所述处理器之间互相通信连接，所述存储器中存储有计算机指令，所述处理器通过执行所述计算机指令，从而执行权利要求1-10任一项所述的多模态神经生物信号的处理方法。A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for processing multimodal neural biological signals according to any one of claims 1 to 10 by executing the computer instructions.
  13. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使计算机执行权利要求1-10任一项所述的多模态神经生物信号的处理方法。A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for processing multimodal neural biological signals described in any one of claims 1-10.
PCT/CN2022/134058 2022-11-24 2022-11-24 Multimodal neural biological signal processing method and apparatus, and server and storage medium WO2024108483A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/134058 WO2024108483A1 (en) 2022-11-24 2022-11-24 Multimodal neural biological signal processing method and apparatus, and server and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2022/134058 WO2024108483A1 (en) 2022-11-24 2022-11-24 Multimodal neural biological signal processing method and apparatus, and server and storage medium

Publications (1)

Publication Number Publication Date
WO2024108483A1 true WO2024108483A1 (en) 2024-05-30

Family

ID=91194971

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/134058 WO2024108483A1 (en) 2022-11-24 2022-11-24 Multimodal neural biological signal processing method and apparatus, and server and storage medium

Country Status (1)

Country Link
WO (1) WO2024108483A1 (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275130A (en) * 2020-02-18 2020-06-12 上海交通大学 Deep learning prediction method, system, medium and device based on multiple modes
CN111461176A (en) * 2020-03-09 2020-07-28 华南理工大学 Multi-mode fusion method, device, medium and equipment based on normalized mutual information
US20200279156A1 (en) * 2017-10-09 2020-09-03 Intel Corporation Feature fusion for multi-modal machine learning analysis
CN113870259A (en) * 2021-12-02 2021-12-31 天津御锦人工智能医疗科技有限公司 Multi-modal medical data fusion assessment method, device, equipment and storage medium
CN114041795A (en) * 2021-12-03 2022-02-15 北京航空航天大学 Emotion recognition method and system based on multi-modal physiological information and deep learning
WO2022149696A1 (en) * 2021-01-06 2022-07-14 주식회사 뷰노 Classification method using deep learning model

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200279156A1 (en) * 2017-10-09 2020-09-03 Intel Corporation Feature fusion for multi-modal machine learning analysis
CN111275130A (en) * 2020-02-18 2020-06-12 上海交通大学 Deep learning prediction method, system, medium and device based on multiple modes
CN111461176A (en) * 2020-03-09 2020-07-28 华南理工大学 Multi-mode fusion method, device, medium and equipment based on normalized mutual information
WO2022149696A1 (en) * 2021-01-06 2022-07-14 주식회사 뷰노 Classification method using deep learning model
CN113870259A (en) * 2021-12-02 2021-12-31 天津御锦人工智能医疗科技有限公司 Multi-modal medical data fusion assessment method, device, equipment and storage medium
CN114041795A (en) * 2021-12-03 2022-02-15 北京航空航天大学 Emotion recognition method and system based on multi-modal physiological information and deep learning

Similar Documents

Publication Publication Date Title
CN109447183B (en) Prediction model training method, device, equipment and medium
Shoeibi et al. Applications of deep learning techniques for automated multiple sclerosis detection using magnetic resonance imaging: A review
US20180144214A1 (en) Deep learning medical systems and methods for image reconstruction and quality evaluation
US11693071B2 (en) Systems and methods for mapping neuronal circuitry and clinical applications thereof
Yoon et al. Medical image analysis using artificial intelligence
CN114748053A (en) fMRI high-dimensional time sequence-based signal classification method and device
CN112690777A (en) Neurological disorder diagnosis system based on state transition dynamic brain network algorithm
KR102427709B1 (en) Multi-modality medical images analysis method and apparatus for diagnosing brain disease
Adams et al. Prediction of outcome in Parkinson’s disease patients from DAT SPECT images using a convolutional neural network
WO2019172181A1 (en) Diagnosis support device, program, learned model, and learning device
Maiti et al. Automatic detection and segmentation of optic disc using a modified convolution network
Wein et al. Forecasting brain activity based on models of spatiotemporal brain dynamics: A comparison of graph neural network architectures
Wang et al. Multiclassification for heart sound signals under multiple networks and multi-view feature
Sharma et al. Classification of heart disease from MRI images using convolutional neural network
WO2024108483A1 (en) Multimodal neural biological signal processing method and apparatus, and server and storage medium
Sonia et al. Cardiac abnormalities from 12‐Lead ECG signals prediction based on deep convolutional neural network optimized with nomadic people optimization algorithm
Das et al. Optimized CT-MR neurological image fusion framework using biologically inspired spiking neural model in hybrid ℓ1− ℓ0 layer decomposition domain
US20220044454A1 (en) Deep reinforcement learning for computer assisted reading and analysis
CN115730269A (en) Multi-modal neurobiological signal processing method, device, server and storage medium
Ferrari Artificial Intelligence for Autism Spectrum Disorders
Çelebi et al. An emotion recognition method based on EWT-3D–CNN–BiLSTM-GRU-AT model
CN114140487A (en) Brain image segmentation method and device
Li et al. An Assisted Diagnosis of Alzheimer's Disease Incorporating Attention Mechanisms Med-3D Transfer Modeling.
CN117788897B (en) Brain age prediction method, device, equipment and storage medium
Saxena et al. FMDB Transactions on Sustainable Computer Letters

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22966178

Country of ref document: EP

Kind code of ref document: A1