CN116110578A

CN116110578A - Screening device for diagnosis of depression symptoms assisted by computer

Info

Publication number: CN116110578A
Application number: CN202211678577.9A
Authority: CN
Inventors: 辜雅婷; 倪士光; 张迟; 马飞; 贾晓健
Original assignee: Shenzhen International Graduate School of Tsinghua University
Current assignee: Shenzhen International Graduate School of Tsinghua University
Priority date: 2022-12-26
Filing date: 2022-12-26
Publication date: 2023-05-12

Abstract

The invention discloses a screening device for computer-aided depression symptom diagnosis, which comprises: the data acquisition module is used for acquiring basic information of a subject and multi-mode data in an experimental interaction task; the data analysis module is used for preprocessing the multi-mode data, extracting multi-mode data characteristics, predicting the multi-mode data characteristics through a constructed depression symptom screening model, and outputting depression symptom prediction classification grading results according to model output results; and the feedback module is used for displaying the multi-mode data characteristic visualization result and the model output result. According to the invention, through the visualization of the real-time data characteristics and the display of the abnormal data, the interpretation of the characteristics can be increased, and the effective screening and recognition of depression symptoms can be realized.

Description

Screening device for diagnosis of depression symptoms assisted by computer

Technical Field

The invention relates to the field of depression identification, in particular to a screening device for computer-aided depression symptom diagnosis.

Background

Depression is caused by many factors such as physiological, psychological and social factors, and belongs to one of typical psychological disorders, and clinical features include significant and persistent low mood, loss of interest and lack of energy, which create a continuous disease burden on the patient himself, home and society. Due to the cognitive prejudice of society to depression, the recognition rate and the visit rate of depression are still at a low level due to insufficient knowledge of patients themselves and families to psychological diseases.

The scale is used as one of the main screening tools of the current depression symptoms, comprises a Beck depression self-rating scale, a PHQ-9 depression screening scale, a Hamilton depression scale and the like, and can be used for evaluating whether the evaluator has depression symptoms and the depression degree. However, due to social impersonation phenomenon in psychological measurement, patients tend to evaluate themselves in a positive manner that can be approved by society, and underestimate negative behaviors that are not accepted by society, so that deviation occurs in evaluation results.

Due to the cognitive prejudice of society to depression, the insufficient knowledge of patients and families to psychological diseases, so that the popularization rate, recognition rate and diagnosis rate of depression are still low, and certain problems and challenges still exist in the application of traditional depression measuring tools and computer-aided intelligent psychological measuring algorithms, including the problems of social imperative errors of measuring tools, high subjective judgment dependence of diagnosis and the like.

Disclosure of Invention

The invention aims to solve the problem of how to improve the recognition rate of depression, and provides a screening device for computer-aided depression symptom diagnosis.

The technical problems of the invention are solved by the following technical scheme:

A screening device for computer-aided depression symptom diagnosis, comprising:

the data acquisition module is used for acquiring basic information of a subject and multi-mode data in an experimental interaction task;

the data analysis module is used for preprocessing the multi-modal data, extracting multi-modal data characteristics, predicting the multi-modal data through a constructed depression symptom screening model, and outputting a depression symptom prediction classification grading result according to a model output result;

and the feedback module is used for displaying the multi-mode data characteristic visualization result and the model output result.

In some embodiments, the data acquisition module comprises a basic information statistics unit, an experimental interaction task unit and a multi-modal data acquisition unit;

the basic information statistics unit is used for collecting the basic information, wherein the basic information comprises basic personal conditions and psychological health conditions;

the experiment interaction task unit is used for guiding the subject to complete the experiment interaction task, and the experiment interaction task comprises the following steps: watching and describing emotion pictures, reading emotion tendencies articles, and semi-structured interviews;

the multi-modal data acquisition unit is used for acquiring multi-modal data in the experimental interaction task unit;

The multi-modal data includes: video frame image data, audio data, and text data.

In some embodiments, the data analysis module comprises: the system comprises a data preprocessing unit, a multi-mode feature extraction unit, a depressive symptom identification unit and a depressive symptom classification and grading unit;

the data preprocessing unit is used for preprocessing the multi-mode data;

the multi-modal feature extraction unit is used for carrying out feature extraction and feature fusion on the preprocessed multi-modal data to obtain multi-modal data features;

the multi-modal data features include: face key feature data, voice frequency energy data and text length word frequency statistical data;

the depression symptom identification unit is used for predicting the multi-mode data characteristics through the constructed depression symptom screening model;

the depressive symptom classification and grading unit is used for outputting a depressive symptom prediction classification and grading result according to the model output result.

In some embodiments, the data analysis module further comprises: the device comprises a characteristic visualization unit and an abnormal data detection and display unit;

the characteristic visualization unit is used for visualizing the multi-modal data characteristics in real time to obtain the multi-modal data characteristic visualization result;

The abnormal data detection and display unit is used for carrying out real-time abnormal detection on the multi-mode data characteristics, obtaining abnormal data detection results and displaying the abnormal data detection results.

In some embodiments, the data analysis module for real-time visualization of multi-modal data features and abnormal data detection and display includes: extracting features according to the differences of the subjects on language expression features, facial behavior features and language mode features, respectively extracting key points from facial image data to serve as primary features, and obtaining blink frequency and smile intensity to serve as secondary features according to eye key points and mouth key points; extracting the MFCC, energy, loudness, zero crossing rate and Mel spectrogram from the audio data; the text data is subjected to text length statistics, LIWC and depression vocabulary rule hit, multi-mode data characteristics are visualized in real time, data exceeding a normal threshold range are displayed abnormally, and abnormal indexes and abnormal occurrence time points are recorded.

In some embodiments, the data analysis module further comprises: a depression recognition model training unit and a model effect verification unit; the multi-modal data includes training data and predictive data;

The depression recognition model training unit is used for dividing the multi-mode data characteristics into a training set, a testing set and a verification set, and training a machine learning model and a deep neural network model in the depression symptom screening model;

the model effect verification unit is used for verifying the trained depression symptom screening model effect and comprises indexes such as model accuracy, recall rate and the like obtained through calculation according to a test set;

the training data is used for training a depression symptom screening model;

the predictive data is used to predict a depressive symptom classification outcome for the subject based on a trained depressive symptom screening model.

In some embodiments, the depression identification model training unit trains a depression symptom screening model comprising: according to the differences of language expression features, facial behavior features and language mode features of depression patients and healthy people, extracting key features for classification prediction, extracting single-frame face pictures and audio from videos, converting texts by an audio automatic voice recognition method, wherein the extracted key features comprise: human face key points, key point secondary characteristics, audio Mel frequency spectrum coefficients, mel frequency spectrograms and other characteristics, text language exploration and word counting.

In some embodiments, the depressive symptom screening model is constructed based on the multimodal data features, classified by a machine learning classification model, a deep neural network model, predicting whether there are depressive symptoms and classifying the depressive symptoms.

In some embodiments, the feedback module includes a client and a server;

the client is used for displaying device diagnosis results, including feature visualization results and depression symptom screening model output results;

the server side is used for inputting the diagnosis result of the doctor into the depression symptom screening model as a real label, and retraining and parameter updating are carried out on the depression symptom screening model.

In some embodiments, after the depression symptom screening model receives the image feature data and the original audio data transmitted by the client through the server, statistical analysis is performed on the data, the depression symptom screening model outputs prediction results and confidence degrees on different modes, the statistics of the deviation from the normal values are marked as abnormal indexes, and the feature visualization results and the model output results are transmitted to the client together as analysis module output results.

Compared with the prior art, the invention has the beneficial effects that:

According to the screening device for assisting in diagnosis of the depression symptoms, the data acquisition module is used for acquiring the multi-modal data, the data analysis module is used for preprocessing the multi-modal data, the characteristics are extracted to obtain multi-modal data characteristics, and the depression symptom prediction classification result is obtained through the depression symptom screening model according to the multi-modal data characteristics; the data analysis module also generates a multi-mode data statistical feature visualization result according to the multi-mode data; the real-time data feature visualization and abnormal data display can increase the interpretability of the feature; the effective screening and identification of depression symptom diagnosis are realized.

The beneficial effects in some embodiments are as follows:

according to the invention, the multi-modal feature extraction unit is used for extracting facial behavior features, language expression features, language mode features and other dimensions to extract symptom feature marks, so that the visualization of real-time data features and the display of abnormal data are realized, and the interpretability of the features is further increased.

The invention helps doctors to carry out auxiliary diagnosis, and the feedback results of the doctors are used for updating training samples of the model and optimizing model parameters.

The invention extracts the multi-mode data characteristics from the multi-mode data, establishes a depression screening model, adopts the client/server side architecture deployment, can assist doctors in diagnosing depression symptoms and depression degree, and has the characteristics of rapidness, low cost, standardization and the like.

Drawings

FIG. 1 is a schematic diagram of a screening apparatus for assisting in diagnosis of depressive symptoms according to an embodiment of the present invention;

FIG. 2 is a flow chart of a data acquisition module in an embodiment of the invention;

FIG. 3 is a flow chart of video processing of audio and video data in an embodiment of the invention;

FIG. 4 is a flow chart of audio processing of audio-visual data in an embodiment of the invention;

FIG. 5 is a training flow chart of a depressive symptom screening model in an embodiment of the present invention;

FIG. 6 is a flow chart of a feedback module in an embodiment of the invention;

FIG. 7 is a training flow chart of the data analysis module in an embodiment of the invention;

FIG. 8 is a flow chart of data analysis module prediction in an embodiment of the invention;

Detailed Description

The invention will be further described with reference to the following drawings in conjunction with the preferred embodiments. It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other.

It should be noted that, in this embodiment, the terms of left, right, upper, lower, top, bottom, etc. are merely relative terms, or refer to the normal use state of the product, and should not be considered as limiting.

With the development of artificial intelligence and other technologies, a computer-aided screening method taking a machine learning algorithm as a core is used for predicting new data by extracting effective marker features and constructing a classification regression model, so that a certain breakthrough is achieved in the biomedical field, and the method can be applied to the field of depression identification.

The embodiment of the invention provides a screening device for computer-aided depression symptom diagnosis, which comprises a data acquisition module, a data analysis module and a feedback module, wherein the device is used for acquiring audio and video data of subjects in psychological experiments and semi-structured interviews, displaying psychological characteristic index information related to the subjects on interactive equipment in real time, generating depression symptom screening reports after evaluation, sending the depression symptom screening reports to doctors and subject clients for computer-aided diagnosis, predicting whether potential depression crowds have depression symptoms, the risk degree of depression and other indexes, and helping the depression risk crowds to carry out extensive and objective auxiliary screening.

Examples:

certain problems and challenges still exist in the application of traditional depression measuring tools and computer-aided intelligent psychological measuring algorithms, including the problems of social impermissibility errors of the measuring tools, high dependence of diagnosis on subjective judgment and the like, and the problems are specifically expressed as follows: (1) Current depression measurement tools rely on reasonable assessment of the subject's own, but due to the social imperceptibility phenomenon prevalent in psychological measurements, i.e. the patient's assessment of the subject's own tends to deviate from negative behaviors that are not acceptable to society, to positive behaviors that can be approved by society, resulting in deviations in the results of psychological assessment; (2) The diagnosis and screening method for depression is complicated, the subjective judgment and the dependence on the diagnosis of a diagnostician are high, the diagnosis and screening method depends on the inquiry of the diagnostician on the psychological condition of a patient and the analysis of clinical manifestation, and is easily influenced by factors such as clinical experience, communication skills, diagnosis and treatment capability and the like of the diagnostician.

The problems of insufficient and unbalanced medical and health resources and the like still exist, so that the current wide screening is difficult, and screening tools and diagnostic modes have certain limitations. Aiming at the problems, the embodiment of the invention provides a rapid and low-cost screening device for diagnosing the symptoms of depression by using a computer.

The embodiment of the invention discloses a computer-aided depression symptom diagnosis screening device which comprises a data acquisition module, a data analysis module and a feedback module. The data acquisition module is used for acquiring video image, voice and text multi-mode data. Wherein the multi-modal data may be divided into training data and predictive data according to purposes; the training data is used for training a depression symptom screening model; the predictive data is used to predict a subject's depressive symptom classification grading result based on the trained depressive symptom screening model. Wherein the training data is multimodal data obtained by collecting a subject who has been diagnosed with depression or health through a data collection module; the prediction data is multi-modal data obtained by collecting the subject to be predicted by the data collection module.

The data analysis module comprises model training and model prediction, and is used for constructing a high-quality data set training high-accuracy depression symptom screening model, predicting new sample data through the constructed model, and outputting a new sample depression symptom classification grading result through the model prediction result. The feedback module comprises a client and a server and is used for displaying the characteristic visualization result and the model output result, and model retraining and model parameter updating and optimizing are carried out according to feedback. The embodiment of the invention extracts the depression symptom mark features from the training data, establishes a computer-aided depression screening model, adopts the client-side/server-side architecture deployment, can assist doctors in diagnosing depression symptoms and depression degree, has the characteristics of rapidness, low cost, standardization and the like, and realizes effective screening and identification of depression symptoms.

The specific implementation is as follows:

the embodiment of the invention provides a screening device for assisting in diagnosis of depression symptoms, which comprises the following components:

the data acquisition module is used for acquiring video image, voice and text multi-mode data;

the data acquisition module comprises a basic information statistics unit, an experiment interaction task unit and a multi-mode data acquisition unit; the basic information statistics unit collects basic conditions and psychological health conditions of individuals, including gender, age and patient health questionnaires (PHQ-9) and scale questionnaires compiled by the generalized anxiety scale (GAD-7); the experimental interaction task unit comprises the steps of watching and describing emotion pictures, reading emotion tendency articles and semi-structured interviews; the multi-mode data acquisition unit acquires audio and video data in the experimental interaction task unit and is further divided into video frame image data, audio data and text data;

the data analysis module in the embodiment is used for constructing a high-quality data set training high-accuracy depressive symptom screening model, and predicting new sample data through the constructed depressive symptom screening model;

the data analysis module comprises model training and model prediction;

the model training is to train a depression symptom screening model with high accuracy by constructing a data set of high-quality training data, and save the trained model structure and parameters; model prediction is performed by inputting new sample data (i.e., prediction data) through the constructed model, and outputting a model output result.

In specific implementation, the model training extracts key features for classification prediction according to differences of language expression features, facial behavior features and language mode features of depression and healthy people, single-frame face pictures and audio are extracted from videos, texts are converted by an audio automatic speech Recognition technology (ASR, automaticSpeech) and extracted features comprise facial key points and key point secondary features, audio Mel frequency spectrum coefficients (MFCC), mel-frequency CepstralCoefficients) and text language exploration and word counting (LIWC, linguisticInquiryandWordCount) and the like.

In specific implementation, the model prediction builds an algorithm model through multi-mode data features of prediction data, the algorithm model integrates feature vectors extracted from the text, audio and image data, classification is carried out through a machine learning classification algorithm and a deep neural network model, and whether depression symptoms exist or not and classification of the depression symptoms are predicted.

The feedback module in this embodiment is configured to display the visualized result of the feature data and the output result of the model, and perform model training and parameter updating.

The feedback module comprises a client and a server;

the client is used for displaying device diagnosis results, including feature visualization results and model output results; the server side is used for inputting the diagnosis result of the doctor into the depression symptom screening model as a real label, and carrying out model retraining and parameter updating on the depression symptom screening model.

In the specific implementation, feature data visualization extracts features according to differences of depressed patients on language expression features, facial behavior features and language mode features, extracts primary features such as key points from facial image data respectively, and further calculates secondary features such as blink frequency, smile intensity and the like according to eye key points and mouth key points; extracting the characteristics of MFCC, energy, loudness, zero crossing rate, mel spectrogram and the like from the audio data; the text data is subjected to text length statistics, LIWC, depression word list rule hits and the like, the characteristic data are visualized in real time, data exceeding a normal threshold range are displayed abnormally, and abnormal indexes and abnormal occurrence time points are recorded.

Server-client feedback module: and the server-side model prediction result is transmitted to the client side, so that a doctor is helped to carry out auxiliary diagnosis screening. Client-server feedback module: the final diagnosis result of the client-side doctor is transmitted to the server-side, and the data and the doctor labeling result help the server to optimize the model parameter improvement model;

in the specific implementation, the model output result is that after the image feature data and the original audio data transmitted by the client are received by the model output result through the server, the data are further statistically analyzed, the predicted result and the confidence coefficient of the model output on different modes are output through the model, the statistic of the deviation from the normal value is marked as an abnormal index, and the feature visualization result and the model output result are jointly transmitted to the client as the analysis module output result.

The model output result can also be called an index analysis result, and is a classification index of depression obtained after screening, namely a classification index of 0/1 and classification indexes of non-depression, mild depression, moderate depression and major depression.

The feature visualization result may also be referred to as a data real-time visualization result or a multi-mode data statistics feature visualization, that is, a data real-time visualization, including a real-time visualization of the features such as the face AU unit, eye gaze direction movement, audio mel spectrogram, word frequency statistics, and the like, and a real-time display of the abnormal data exceeding the threshold.

In the specific implementation, the model retraining is carried out according to the auxiliary diagnosis result of a final device of a user by a doctor as a label, the newly acquired audio and video characteristic data are added into a database of training data as new sample data (i.e. predicted data after prediction), and all samples are retrained at regular time to iteratively update model parameters, so that the model accuracy is improved.

Experimental example:

the experimental example provides a screening device for assisting in diagnosis of depression symptoms, which is shown in fig. 1, and comprises a data acquisition module 11, data analysis modules 12 and 13 and feedback modules 14 and 15, wherein depression symptom sign features are extracted from multi-mode data, a depression symptom screening model is established, and a client/server side architecture is adopted to assist a doctor in diagnosing depression symptoms and depression degree, so that the screening device has the characteristics of rapidness, low cost, standardization and the like.

The data acquisition module 11 is used for acquiring video image, voice and text multi-modal data, and comprises a basic information statistics unit 101, an experiment interaction task unit 102 and a multi-modal data acquisition unit 103.

The basic information statistics unit 101 collects basic personal conditions and mental health conditions, including gender, age, and patient health questionnaires (PHQ-9) and scale questionnaires compiled by the generalized anxiety scale (GAD-7) (i.e., filling in a depression screening scale), and experimental results will be used as references for whether or not the subjects have depression symptoms and the extent of the depression symptoms, and the subjects with depression need to be diagnosed by doctors in the model training stage.

The experimental interaction task unit 102 performs experimental task interactions including viewing and describing emotion pictures, reading emotion tendencies articles, semi-structured interviews, as shown in fig. 2, specifically including:

step S201, selecting three types of pictures of negative (titer 1.4-3.6), neutral (titer 3.7-5.9) and positive (titer 6.0-7.8) from Chinese emotion material emotion picture system (CAPS) as stimulus to induce emotion expression, selecting 10 pictures from each type, and displaying each picture on a screen for 3 seconds in an experimental process, wherein each picture is randomly displayed once.

Step S202, the subject needs to describe a representative mood map at the end of each group of pictures.

In step S203, text segments with positive emotion tendencies and negative emotion tendencies are selected from the articles with emotion tendencies, and the subject needs to read the text on the screen aloud at the normal speaking speed.

Step S204, randomly selecting positive, neutral, and negative face pictures from the chinese facial emotion image system (CFAPS) as stimulus to induce emotion expression, and presenting 3 pictures separately to the subject in random order, requiring the subject to describe the content of the pictures as much as possible. The subject evokes an emotional response by viewing pictures of different emotions, reading articles with emotional tendencies, and facial pictures describing different emotions.

Step S205, semi-structured interviews are performed by the dummy on the subjects according to questions in the interview question list, the interview content including 18 active, neutral and passive trend questions selected at any time from 30 questions, 6 for each category.

The multi-mode data acquisition unit 103 records and acquires audio and video data in the experimental interaction task, namely multi-mode data such as voice and video of a subject, through a camera, a microphone and other devices, and is further divided into multiple modes such as video frame pictures, audio and texts.

The data analysis modules 12 and 13 comprise a model training 12 and a model prediction 13, the model training 12 trains a high-accuracy depression symptom screening model by constructing a high-quality data set, and the trained model structure and parameters are saved; the model prediction 13 predicts the input new sample data by the constructed model, and outputs the model output result. The data analysis modules 12 and 13 mainly analyze different mode data such as pictures, voices and texts around symptom marks and recognition features of the depressive patients in the dimensions such as facial behavior features, language expression features and language mode features, and assist in classifying and screening the depression.

As shown in fig. 1 in particular, the data analysis module includes model training 12 and model prediction 13; model training 12 includes a data preprocessing unit 104, a multi-modal feature extraction unit 105, a depression recognition model training unit 106, and a model effect verification unit 107. The model prediction 13 includes the above-described data preprocessing unit 104 and multi-modal feature extraction unit 105, and a depression symptom recognition unit 108, feature visualization unit 109, abnormal data detection and display unit 110, depression symptom classification grading unit 111.

The data preprocessing unit 104 performs preprocessing on the data acquired by the data acquisition module, including deletion segment elimination and extraction of the voice segment of the subject.

The multi-modal feature extraction unit 105 extracts features of the above-described modal data, specifically including face key feature data, voice frequency energy data, text length word frequency statistics data, and the like.

The depression recognition model training unit 106 classifies the extracted feature data into a training set, a test set, and a verification set at a ratio of 10:1:1, trains a machine learning model, and a deep neural network.

The model effect verification unit 107 verifies the trained network effect, including calculating indexes such as model accuracy, recall, and the like according to the test set.

The depression symptom identifying unit 108 predicts new data samples by trained model structures and parameters.

The feature visualization unit 109 performs real-time visualization on the multi-mode data statistics features, including two-dimensional and three-dimensional reconstruction features of key points of the image mode face, eye gaze direction and change features, AU numerical features of the key face unit, and the like; audio mode audio waveform diagram, mel spectrogram features, etc.; text length features of text modalities, word hits features of word lists, etc.

The abnormal data detecting and displaying unit 110 performs real-time abnormal detection on the multi-mode features, including that the time interval between the key points of the face and the gaze direction change of the eyes exceeds a threshold value; AU face units associated with sadness, fear, aversion, etc. have long duration and high intensity; word list hits and the like are identified by depression after text word segmentation.

The depression symptom classification ranking unit 111 outputs 0/1 classification of depression and prediction of mild, moderate, and major depression levels by the above model prediction and rule hit results.

From a behavioral dimension, depressed patients behave differently on head, face, limb movements than non-depressed populations. The amplitude of head movement is reduced and the duration of head drop is prolonged in patients with depression. Facial expression activities are taken as important research contents of emotion recognition, can effectively analyze the emotion states of a subject, and are further applied to the field of depression recognition. The symptoms of depression are embodied as: the facial expression ability of the depressed population is impaired, and sad emotional characteristics such as eye relaxation, frowning and the like are more easily represented. The characteristic differences on the key parts of the human face such as eyebrows, eyes, mouths and the like are obvious, and the characteristic differences comprise eye cavities, eye contact reduction, gazing direction lack change and eye opening and closing times reduction; lack of variation in mouth movement, significantly reduced smile intensity and duration, etc.

According to the facial expression characteristics of the depression patients, the experimental example provides that the processing of the video data specifically comprises the following steps as shown in fig. 3:

in step S301, video data is acquired.

Step S302, obtaining a single-frame picture through video framing.

Step S303, further extracting the facial key features.

In steps S304 to S306, in order to ensure the face privacy of the subject, 68 key points, face units (AU) and eyes of the face region along with the time stamp in the video frame image data are extracted by using OpenFace tool, and only key feature data is reserved when the personal face image information is desensitized in the standard open data set.

S307-S312, extracting secondary features on key points, face units and eye vision of the face, wherein the secondary features comprise motion statistics head motion and amplitude of three-dimensional coordinate points of 68 key points of the face in a coordinate system; calculating the areas of polygons formed by the key points of the left eyes 36-41 and the key points of the right eyes 42-47 respectively, and analyzing the opening and closing times of eyes of a subject, so as to further count the blink frequency of the eyes; calculating the curvature of a curve formed by key points of the lower lip 54-60 of the mouth part, and analyzing the times and the mouth smile intensity of the subject smile; drawing the change of the intensity of the face unit of the human face along with time, and analyzing the emotion and emotion intensity of a subject such as happiness (superposition of AU6 and AU 12), sadness (superposition of AU1, AU4 and AU 15) and the like; calculating the change angle of the eye sight direction vector, and analyzing the sight change angle; and counting the positions of the sight lines concentrated on the screen according to the eye gaze vectors after the screen sight line calibration, and analyzing the sight line attention area of the subject.

In the speech expression of the depression patients, the characteristics of slow response, thought blocking, poor language fluency and the like are often presented, and the voice characteristics of slow speech speed, low voice, lack of tone change, increase of pause times and single pause duration and the like are presented. From the linguistic dimension, depressed patients have characteristics of negative mood bias, psychological rumination, self-attention and the like, and are particularly characterized by obvious negative linguistic expression, excessive attention to self-body abnormality and the like.

According to the language expression characteristics of the depressed patient, the experimental example provides that the processing of the audio data specifically comprises the following steps as shown in fig. 4:

in step S401, audio data is acquired.

Step S402, extracting a speech segment of each answer of the subject from the audio data.

Step S403, merging the complete voice data of the single subject, and segmenting the voice fragments of 10S, 30S, 60S.

Step S404, frame windowing and fast fourier transform (FFT, fastFourier transform) are performed on the voice data.

Step S405, unifies the sampling frequency of the voice signal to 16000Hz, obtains text data through ASR translation, and manually corrects the translated text.

Steps S406-S408, extracting acoustic features such as low-level descriptors (LLD, lowLevelDescriptor) in the audio frequency and the like, including prosodic features such as energy, loudness, formants, zero-crossing rate and the like, aiming at voice data through an OpenSmile tool; spectral features such as MFCC, psychoacoustic clarity, spectral variance, skewness, kurtosis, etc.; fundamental frequency F0, tone quality, fundamental frequency perturbation Jitter, amplitude perturbation Shimmer and other sound characteristics; and the secondary statistical characteristics of the feature quartile, the maximum and minimum values, the contour centroid, the linear prediction gain and the like.

And S409-S411, extracting characteristics associated with depression symptom marks in the text for analysis of depression according to the characteristics of depression language use modes, wherein the characteristics comprise text length analysis, word frequency statistical analysis and emotion tendency analysis. According to the characteristics of speech reduction, incomplete expression and the like of a depressed patient, carrying out statistical analysis on the sentence length of a text fragment in a subject experiment; generating a co-occurrence network of the depressed patient and the healthy control text segment according to the characteristics of increased first person expression, repeated negative language expression and the like of the depressed patient, wherein the size of a network node represents the occurrence frequency of the word or the word, filtering the text co-occurrence of the depressed and the healthy control, and analyzing important words in the expression of the depressed patient; the vocabulary includes passive vocabulary expressions (dead words, anxiety words, anger words), depressed vocabulary expressions (depression, antidepressants, mental diseases), somatic alterations expressions (headache, insomnia, inappetence), and emotional tendency is analyzed by vocabulary hits.

Based on the existing data set, the experimental example provides a computer-aided screening method taking a machine learning algorithm as a core, extracts characteristics for classification to conduct classification prediction on new data samples, and carries out aided identification on depression symptoms. According to differences of the depressed patient in image, voice, text and other modal data, respectively establishing a depression screening model from the three modalities, wherein the specific steps include as shown in fig. 5:

Step S501, acquiring audio and video data acquired in an interaction task.

Step S502, extracting picture data from the audio/video data.

Step S503, extracting features such as face key points, AU units, and sight lines from the picture data under different time stamps.

Step S504, inputting the characteristics into a balanced random forest (BRF, balancedRandomForest) model for parameter adjustment. The BRF randomly samples data with more categories through a clustering-based downsampling method, so that depressed and healthy control samples with balanced sample size are obtained and input into a decision tree for parameter adjustment, and model results are aggregated at an output layer, so that prediction deviation caused by the fact that models are stressed by unbalanced depressed samples and healthy samples can be solved.

Step S505 extracts audio data from the audio-video data.

In step S506, features such as MFCC, mel-frequency spectrogram, etc. are extracted from the audio data.

Step S507, randomly sampling, and inputting the equalized sample characteristics into a Convolutional Neural Network (CNN) and a bidirectional long and short time memory network (BiLSTM) model for training. The CNN model consists of three layers of convolutions, dropouts and max pooling, connecting the flat layer and full connection layer outputs. The BiLSTM network is composed of two layers of bidirectional cyclic neural networks, and is connected with the output of the Flatten layer and the output of the full-connection layer. The BiLSTM adopts a mode of combining forward feature extraction and reverse feature extraction, can be connected with context information, is suitable for modeling a time sequence model, and can extract more comprehensive audio data features.

In step S508, text data is extracted and translated from the audio/video data, and the translated text is manually corrected.

Step S509 extracts features such as text LIWC and vocabulary matching from the text data.

Step S510 applies to the downstream depression identification classification by fine tuning parameters on the Bert and Erine pre-training models. The original data is input into a pre-training model after being instantiated, training is carried out according to default configuration, training is carried out through text data of marked depressed subjects and health control, and a text classification result of a test set is output.

Step S511, weight is distributed to the effectiveness of the judging results of the three models through automatic weight learning, and depression symptom prediction is output.

The feedback module comprises a server side and a client side. The server side consists of two parts of model training 61 and model updating 62, and the client side consists of two parts of a subject 63 and a doctor 64;

as shown in fig. 1, the feedback module includes a client 14 and a server 15; the client 14 includes a feedback report generating unit 112, a physician diagnosing unit 113; the server 15 includes a training sample storage unit 114 and a model update feedback unit 115;

the feedback report generating unit 112 presents a feedback prediction result and a key feature identification screening result on the client in a chart visualization mode according to the abnormal data detection and display and depression symptom classification result;

The doctor diagnosis unit 113 calculates a risk level and a confidence level according to the feedback report result, recommends a doctor to diagnose for a subject with a higher risk level, and other subjects select whether to diagnose the doctor according to personal conditions;

the training sample storage unit 114 distributes data labels according to the auxiliary diagnosis result of the doctor device, stores the data labels and the original data characteristics into a new training sample database, and performs model retraining and parameter updating;

the model update feedback unit 115 performs periodic training update on all stored samples, and replaces the original model to serve as a new prediction model under the condition that the comparison index of the new model and the old model is improved.

As shown in fig. 6, the method specifically comprises the following steps:

step S601, storing a model training sample.

Step S602, performing model training.

And step S603, model prediction is carried out on the new sample through training the verified model structure and parameters.

In step S604, the server side predicts the new sample and then feeds back the visualized result of the feature data and the output result of the model to the subject and the physician client.

In step S605, the doctor performs diagnosis in combination with the prediction result.

In step S606, the physician feeds back the diagnosis result of the physician to the server as a real label to re-input the model to assist in model optimization.

In step S607, if the prediction result is accurate, the new sample is used as a training sample of the newly added model to perform model iterative training and parameter updating within a certain time.

Step S608, if the prediction result is inaccurate, the error sample weight is increased, and the network update model is retrained.

The experimental example comprises two steps of model parameter training and new sample prediction, as shown in fig. 7 and 8, algorithm training is carried out to train a high-accuracy depression symptom screening model through an acquired data set, and the trained model structure and parameters are saved; and (3) inputting multimode data samples of the potential crowd of depression for prediction through the constructed model by algorithm prediction, and outputting characteristic data visualization and model output results.

Specifically, the model training extracts a corresponding feature training model according to the experimental acquisition data, and saves the model structure and model parameters, which specifically includes:

in step S701, the subject signs an informed consent claim, fills in information such as gender and age, and defines an experimental procedure.

Step S702, randomly selecting positive, neutral and negative pictures from Chinese emotion material emotion picture system (CAPS) and Chinese facial emotion image system (CFAPS), selecting a segment with 99% positive trend and a text segment with 97% negative trend from articles, and collecting audio and video data of the subject by guiding the subject to watch experimental tasks such as describing emotion pictures, reading emotion articles, interviewing face to face, and the like.

In step S703, whether the subject suffers from depression is determined by the diagnosis of doctor and the total score of PHQ-9, the subject data label of diagnosis of depression is 1, and the subject data label of healthy control is 0.

Step S704, carrying out batch preprocessing on the data, deleting the missing fragments and the invalid fragments, and extracting the audio and text data of the subject.

Step S705, determining differences in language expression features, facial behavior features, and language usage patterns of depressed patients, and extracting features capable of distinguishing between depression and healthy controls, specifically comprising: 1) Extracting primary characteristics such as face key points, 0/1 value and intensity of an AU unit of a face of a patient, eye gaze direction and the like according to the performance of the patient on the face, and calculating secondary characteristic data such as blink times, smile intensity, eye gaze angle change and the like according to the characteristic performance of the depressed patient on eyebrows, eyes and lips; 2) Extracting energy, loudness, MFCC, mel spectrogram and the like aiming at the characteristic expression of a patient on language expression; 3) Aiming at the characteristic expression of a patient on a language use mode, analyzing characteristics such as word frequency of a text through a LIWC and other natural language processing analysis tools.

Step S706, a machine learning model is built, wherein the machine learning model comprises an image feature BRF balanced random forest, an audio feature bidirectional long and short time memory network BiLSTM and a text feature Erine pre-training model.

Step S707, the model effect is verified by dividing the test set and the new sampling test data, and the trained model structure and parameters are saved at the server side for classification of the new data.

Specifically, the model prediction predicts whether the newly collected subjects have depressive symptoms and grading of the depressive symptoms according to the model structure and parameters saved by training, and specifically includes:

step S801, prompting the testee to watch and describe the experimental flows of emotion pictures, reading emotion articles, semi-structure interviews and the like through characters and virtual characters displayed on an interactive display screen.

Step S802, collecting audio and video data of a subject in the process through a camera and a microphone.

Step S803, extracting video frame pictures, audio and text obtained by converting audio ASR in the audio and video data, respectively preprocessing, extracting audio and text data of a subject, and preprocessing.

Step S804, extracting single frame pictures from the video, extracting features of each frame picture by an Openface tool, wherein the features comprise eye gaze direction, face key points, AU face unit 0/1 value and intensity, and secondary features such as blink frequency, smile intensity, and the like; extracting a subject voice fragment from a video, and extracting the characteristics of MFCC, mel spectrograms and the like through an OpenSmile tool; text is obtained from the voice fragments through ASR translation, and features such as word frequency, emotion tendency and the like of the text are analyzed through a LIWC and other natural language processing analysis tools.

Step S805, inputting the obtained features into a stored model, and predicting a new subject according to different modal feature data, including BRF balanced random forest, audio feature bidirectional long-short-time memory network BiLSTM and text feature Erine pre-training model.

Step S806, the above features are visualized in real time at the client.

Step S807-S809, the face features include two-dimensional and three-dimensional reconstruction features of key points of the face, eye gaze direction and variation features, AU numerical features of key face units, and the like; the audio features S808 include an audio waveform diagram, mel-frequency spectrogram features, and the like; the text feature S809 includes a text length feature, a vocabulary hit feature, and the like.

Step S810, detecting and displaying abnormal data such as audio and video characteristic data exceeding a threshold value or text data hit in a word list in real time, and recording starting and ending time points of the abnormal data.

Step S811, the classifying and grading results of the depression symptoms are transmitted to a client through a server, and the client displays the real-time visual results of the data in the experiment and the final model output results and sends the results to a doctor for auxiliary screening and identification of the depression symptoms.

The following is a description of the beneficial effects of the experimental examples of the invention:

The experimental example combines the technologies of client/server architecture, machine learning and the like, realizes training of a large model at a server, and conveniently carries out interactive experimental tasks at the client. The original data acquired by the interactive experiment is stored locally, so that the face privacy of a patient can be effectively protected; extracting symptom characteristic marks from dimensions such as facial behavior characteristics, language expression characteristics, language mode characteristics and the like of a patient, and increasing the interpretability of the characteristics through real-time data characteristic visualization and abnormal data display; the model prediction report is sent to the subject and the doctor client side simultaneously, so that the doctor is assisted in diagnosis, and the feedback result of the doctor is used for updating the training sample of the model and optimizing the model. The effective classification of depression diagnosis is realized through a standardized screening process, interpretable identification features, a high-accuracy screening model and server-client bidirectional feedback. The experimental example screening flow standard ensures the interpretability of the screening extraction characteristics and the accuracy of the screening model, and can better predict whether the subject suffers from depression and the severity of the depression.

The foregoing is a further detailed description of the invention in connection with the preferred embodiments, and it is not intended that the invention be limited to the specific embodiments described. It will be apparent to those skilled in the art that several equivalent substitutions and obvious modifications can be made without departing from the spirit of the invention, and the same should be considered to be within the scope of the invention.

Claims

1. A screening apparatus for computer-aided depression symptom diagnosis, comprising:

2. The screening device for computer-aided depression symptom diagnosis of claim 1, wherein the data acquisition module comprises a basic information statistics unit, an experimental interaction task unit and a multi-modal data acquisition unit;

3. The screening apparatus for computer-aided depression symptom diagnosis of claim 1, wherein the data analysis module comprises: the system comprises a data preprocessing unit, a multi-mode feature extraction unit, a depressive symptom identification unit and a depressive symptom classification and grading unit;

the data preprocessing unit is used for preprocessing the multi-mode data;

4. The screening apparatus for computer-aided depression symptom diagnosis of claim 1, wherein the data analysis module further comprises: the device comprises a characteristic visualization unit and an abnormal data detection and display unit;

5. The screening device for computer-aided depression symptom diagnosis of claim 4, wherein the data analysis module visualizes and detects and displays abnormal data on a multi-modal data feature in real time, comprising: extracting features according to the differences of the subjects on language expression features, facial behavior features and language mode features, respectively extracting key points from facial image data to serve as primary features, and obtaining blink frequency and smile intensity to serve as secondary features according to eye key points and mouth key points; extracting the MFCC, energy, loudness, zero crossing rate and Mel spectrogram from the audio data; the text data is subjected to text length statistics, LIWC and depression vocabulary rule hit, multi-mode data characteristics are visualized in real time, data exceeding a normal threshold range are displayed abnormally, and abnormal indexes and abnormal occurrence time points are recorded.

6. The screening apparatus for computer-aided depression symptom diagnosis of claim 3, wherein the data analysis module further comprises: a depression recognition model training unit and a model effect verification unit; the multi-modal data includes training data and predictive data;

the depression recognition model training unit is used for dividing training data provided by the characteristics into a training set, a testing set and a verification set, and training a machine learning model and a deep neural network model in the depression symptom screening model;

the model effect verification unit is used for verifying the trained depression symptom screening model effect and comprises model accuracy and recall index obtained through calculation according to a test set;

the training data is used for training a depression symptom screening model;

7. The screening apparatus for computer-aided depression symptom diagnosis of claim 6, wherein the depression identification model training unit trains a depression symptom screening model comprising: according to the differences of language expression features, facial behavior features and language mode features of depression patients and healthy people, extracting key features for classification prediction, extracting single-frame face pictures and audio from videos, converting texts by an audio automatic voice recognition method, wherein the extracted key features comprise: human face key points, key point secondary characteristics, audio Mel frequency spectrum coefficients, mel frequency spectrum characteristics, text language exploration and word counting.

8. The screening device for diagnosis of depressive symptoms by computer assistance according to claim 1, wherein the depressive symptom screening model is constructed based on the multimodal data features, classified by a machine learning classification model, a deep neural network model, predicting whether there is a depressive symptom and classifying the depressive symptom.

9. The screening device for computer-aided depression symptom diagnosis of claim 1, wherein the feedback module comprises a client side and a server side;

10. The screening device for diagnosis of depressive symptoms by computer assistance according to claim 9, wherein after the depressive symptom screening model receives the image feature data and the original audio data transmitted by the client through the server, statistical analysis is performed on the data, the predicted results and the confidence levels of the data on different modes are output through the depressive symptom screening model, the statistics of the deviation from the normal values are marked as abnormal indexes, and the feature visualization result and the model output result are transmitted to the client together as the analysis module output result.