CN109350051B - Head wearable device for mental state assessment and adjustment and working method thereof - Google Patents
Head wearable device for mental state assessment and adjustment and working method thereof Download PDFInfo
- Publication number
- CN109350051B CN109350051B CN201811430882.XA CN201811430882A CN109350051B CN 109350051 B CN109350051 B CN 109350051B CN 201811430882 A CN201811430882 A CN 201811430882A CN 109350051 B CN109350051 B CN 109350051B
- Authority
- CN
- China
- Prior art keywords
- mental state
- audio
- module
- state evaluation
- source file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000006996 mental state Effects 0.000 title claims abstract description 167
- 238000000034 method Methods 0.000 title claims abstract description 20
- 238000011156 evaluation Methods 0.000 claims abstract description 102
- 230000008909 emotion recognition Effects 0.000 claims abstract description 15
- 210000003128 head Anatomy 0.000 claims description 46
- 210000001061 forehead Anatomy 0.000 claims description 31
- 230000008451 emotion Effects 0.000 claims description 22
- 210000000988 bone and bone Anatomy 0.000 claims description 13
- 238000013527 convolutional neural network Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 9
- 230000005236 sound signal Effects 0.000 claims description 9
- 238000004891 communication Methods 0.000 claims description 8
- 210000005069 ears Anatomy 0.000 claims description 7
- 230000000638 stimulation Effects 0.000 claims description 6
- 238000000513 principal component analysis Methods 0.000 claims description 4
- 230000002996 emotional effect Effects 0.000 claims description 3
- 238000012880 independent component analysis Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 210000004556 brain Anatomy 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 8
- 230000005611 electricity Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000005484 gravity Effects 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 210000000944 nerve tissue Anatomy 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 210000003625 skull Anatomy 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 230000005489 elastic deformation Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 210000001595 mastoid Anatomy 0.000 description 1
- 230000004630 mental health Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- NJPPVKZQTLUDBO-UHFFFAOYSA-N novaluron Chemical compound C1=C(Cl)C(OC(F)(F)C(OC(F)(F)F)F)=CC=C1NC(=O)NC(=O)C1=C(F)C=CC=C1F NJPPVKZQTLUDBO-UHFFFAOYSA-N 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Psychiatry (AREA)
- Engineering & Computer Science (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Pathology (AREA)
- Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Psychology (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Child & Adolescent Psychology (AREA)
- Developmental Disabilities (AREA)
- Educational Technology (AREA)
- Hospice & Palliative Care (AREA)
- Social Psychology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention provides a head wearable device for mental state assessment and adjustment, characterized in that: comprising the following steps: an audio playing module for playing the audio source file; the physiological signal acquisition module is used for acquiring physiological signals of the tested person; and the mental state evaluation module is used for carrying out emotion recognition and mental state evaluation on the physiological signals acquired by the physiological signal acquisition module to obtain mental state evaluation results and feeding the mental state evaluation results back to the audio playing module to adjust the played audio source file. The device integrates mental state evaluation and intervention adjustment, is convenient to use, and can rapidly and effectively evaluate and adjust the mental state of a tested person. The invention also provides a working method of the head wearable device, which can collect physiological signals of the tested person to evaluate the mental state of the tested person when the tested person receives the audio stimulus, and feed back and adjust the audio playing content in real time so as to adjust the mental state of the tested person in real time.
Description
Technical Field
The present invention relates to the technical field of mental state evaluation and adjustment, and more particularly, to a head wearable device for mental state evaluation and adjustment and a working method thereof.
Background
People are commonly stressed from various aspects such as work, study, life and the like, and the stress easily causes poor mental state, so that hidden danger is brought to mental health of people. In order to obtain mental state information of people, the existing head wearable equipment can acquire physiological signals of people so as to test and evaluate mental states of people, but the existing head wearable equipment has the following defects:
firstly, the existing head wearable equipment can only realize a single function, most of the existing head wearable equipment can only acquire physiological signals of a human body, then data are required to be transmitted to other equipment for analysis and processing to obtain mental state identification results, and mental states cannot be effectively adjusted in real time;
secondly, the existing mental state evaluation system is used for analyzing the emotion of a person by collecting physiological signals of the person, such as signals of brain electricity, skin electricity, myoelectricity, heart rate and the like, so that the mental state is evaluated; however, the existing mental state evaluation system based on brain electricity collects brain electricity signals through electrodes, then carries out data processing by algorithms such as short-time Fourier transform, principal component analysis and the like, extracts data characteristics, and recognizes emotion in different frequency bands, but the characteristics extracted by the algorithm have poor effect when being applied to emotion recognition;
thirdly, the position of the existing head wearable equipment needs to be adjusted when the equipment is worn, so that the wearing is inconvenient; after wearing, the head is not firmly fixed with the head, so that position deviation easily occurs, and physiological signals cannot be effectively acquired, or the head is easy to slide off; the wearing comfort is poor, and the wearing comfort is not suitable for long-time use.
Disclosure of Invention
To overcome the disadvantages and shortcomings in the prior art, an object of the present invention is to provide a head wearable device for mental state evaluation and adjustment, which can collect physiological signals of a subject to evaluate mental state of the subject when the subject receives audio stimulus, and feed back and adjust audio play content in real time, so as to adjust mental state of the subject in real time. The invention also aims to provide a working method of the head wearable device, which can collect physiological signals of a tested person to evaluate the mental state of the tested person when the tested person receives audio stimulus, and feed back and adjust audio play content in real time so as to adjust the mental state of the tested person in real time.
In order to achieve the above purpose, the invention is realized by the following technical scheme: a head wearable device for mental state assessment and adjustment, characterized by: comprising the following steps:
an audio playing module for playing the audio source file;
the physiological signal acquisition module is used for acquiring physiological signals of the tested person;
and the mental state evaluation module is used for carrying out emotion recognition and mental state evaluation on the physiological signals acquired by the physiological signal acquisition module to obtain mental state evaluation results and feeding the mental state evaluation results back to the audio playing module to adjust the played audio source file.
The head wearable device can collect physiological signals of the tested person to evaluate the mental state of the tested person, and feed back and adjust the playing audio in real time to adjust the playing content, so that the emotion of the tested person is influenced in real time, and the mental state is adjusted; the device is convenient to use, and can be used for rapidly and effectively evaluating the mental state and performing intervention adjustment on the tested person.
Preferably, the device further comprises a wearable frame body; the wearable frame body comprises a forehead frame and positioning frames connected to two sides of the forehead frame; the positioning frame comprises an extension rod body connected with the forehead frame and an audio playing seat body connected below the extension rod body; the audio playing seat body and the extension rod body form a positioning part for positioning on human ears; the audio playing module is arranged on the audio playing seat body; the physiological signal acquisition module is an electroencephalogram signal acquisition module; the electroencephalogram signal acquisition module is arranged on the forehead frame; the mental state evaluation module is arranged on the positioning frame; the electroencephalogram signal acquisition module, the mental state evaluation module and the audio playing module are sequentially connected through signals.
According to the head wearable device, the mental state evaluation module and the audio playing module are arranged on the positioning frame, so that the head wearable device is light in weight front and heavy in weight back, the forehead frame is attached to the forehead of a human body under the action of gravity, and the electroencephalogram signal acquisition module can effectively acquire physiological signals of the human body; the locating rack that extends body of rod and audio playback pedestal formed is "T" shape, and the rear side frame of "T" shape is established on human ear, can realize firm, convenient location and fixed, has good wearing comfort simultaneously.
Preferably, the electroencephalogram signal acquisition module comprises a frontal pole central line electroencephalogram acquisition unit arranged on the inner side of the forehead frame and frontal pole electroencephalogram acquisition units which are arranged on the inner side of the forehead frame and are positioned on two sides of the frontal pole central line electroencephalogram acquisition unit. The invention can effectively collect the brain electrical signals from one lead to multiple leads of the human body.
Preferably, the wireless communication device further comprises a wireless communication module; the wireless communication module is arranged on the positioning frame; the mental state evaluation module is in signal connection with the cloud platform through the wireless communication module.
The working method of the head wearable device for mental state evaluation and adjustment is characterized by comprising the following steps of: the method comprises the following steps:
s1, an audio playing module plays an audio source file and performs auditory stimulation on a tested person;
s2, a physiological signal acquisition module acquires physiological signals of a tested person; the physiological signals comprise any one or more than two of brain electrical signals, heart rate signals, electromyographic signals, skin electrical signals and voice signals;
s3, preprocessing, feature extraction and emotion recognition are carried out on the physiological signals by the mental state evaluation module, and mental states of the testee are evaluated to obtain mental state evaluation results;
and S4, the mental state evaluation module feeds back a mental state evaluation result to the audio playing module so as to adjust the audio source file.
Preferably, in step S1, the audio playing module plays the audio source file, and performs auditory stimulation on the tested person, which means that: the audio playing module is a loudspeaker, an in-ear earphone or a bone conduction earphone; the loudspeaker or the in-ear earphone or the bone conduction earphone plays the audio source file to perform auditory stimulus on the tested person.
Preferably, in step S2, the physiological signal acquisition module acquires a physiological signal of the subject, which means that: the physiological signal acquisition module acquires an electroencephalogram signal of a tested person;
in the step S3, the mental state evaluation module performs preprocessing, feature extraction and emotion recognition on the physiological signal, and evaluates the mental state of the tested person to obtain a mental state evaluation result, which means that: the mental state evaluation module is used for preprocessing the electroencephalogram signals, carrying out feature extraction and emotion recognition by adopting a dynamic graph convolution neural network algorithm and combining a width learning system, and evaluating the mental state of the tested person to obtain a mental state evaluation result.
Preferably, the mental state evaluation module performs preprocessing on the electroencephalogram signals, performs feature extraction and emotion recognition by adopting a dynamic graph convolution neural network algorithm in combination with a width learning system, evaluates the mental state of the tested person, and obtains a mental state evaluation result, which means that: removing the artifacts of electro-oculogram, electrocardiographic and electromagnetic interference in the electroencephalogram by adopting an independent component analysis algorithm and a principal component analysis algorithm so as to realize pretreatment; extracting the characteristics of the electroencephalogram signals by adopting a dynamic graph convolutional neural network algorithm, mapping the electroencephalogram signals to a characteristic space, and then adopting a width learning system as a classifier in the characteristic space to carry out emotion classification and identification on the electroencephalogram signals to obtain the intensity values of all emotion categories; and evaluating the mental state of the tested person according to the intensity value of each emotion type.
Preferably, the emotional categories include fatigue, depression, and boring; assessing the mental state of the subject based on the intensity values of the respective emotion categories means: the mental state of the subject is evaluated by setting a fatigue normal value, a depression normal value and a boring normal value, and judging the ratio of the fatigue strength value to the fatigue normal value, the ratio of the depression strength value to the depression normal value and the boring strength value to the boring normal value, respectively.
Preferably, in the step S4, the mental state estimation module feeds back the mental state estimation result to the audio playing module to adjust the audio source file, which means: one of three schemes is adopted:
1. the mental state evaluation module calls out a corresponding audio source file from a play audio library in the mental state evaluation module according to the mental state evaluation result, and sends the audio source file to the audio play module, and the audio play module plays the received audio source file to adjust play audio;
2. the mental state evaluation module generates a corresponding audio signal generation instruction according to the mental state evaluation result and sends the audio signal generation instruction to the audio playing module; the audio playing module calls out a corresponding audio source file from a playing audio library in the audio playing module according to the audio signal instruction to play so as to adjust the playing audio;
3. the mental state evaluation module sends the mental state evaluation result to the external terminal equipment, the external terminal equipment calls out a corresponding audio source file from a playing audio library in the external terminal equipment according to the mental state evaluation result, and sends the audio source file to the audio playing module, and the audio playing module plays the received audio source file to adjust playing audio.
Preferably, in step S4, the mental state estimation module is further in signal connection with the cloud platform through the wireless communication module, so as to send the physiological signal and the mental state estimation result to the cloud platform.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the head wearable device can collect physiological signals of the tested person to evaluate the mental state of the tested person, and feed back and adjust the playing audio in real time to adjust the playing content, so that the emotion of the tested person is influenced in real time, and the mental state is adjusted; the device is convenient to use, and can quickly and effectively evaluate, intervene and adjust the mental state of a tested person;
2. the head wearable device can be firmly and conveniently fixed on the head of a human body, and has good wearing comfort; effectively collecting brain electrical signals from one lead to multiple leads of a human body;
3. the method can collect the physiological signals of the testee to evaluate the mental state of the testee, and feed back and adjust the playing audio in real time to adjust the playing content, thereby affecting the emotion of the testee in real time and adjusting the mental state;
4. in the mental state evaluation module, a dynamic graph convolution neural network algorithm (Dynamical Graph Convolutional Neural Networks, DGCNN) and a width learning system (Broad Learning System, BLS) are adopted, namely, the algorithm of DGCNN+BLS is adopted to realize the evaluation of mental states; the dynamic graph convolution neural network algorithm is an extension of the convolutional neural network algorithm (CNN) on the graph; the research object of the traditional CNN is mainly aimed at the data of the regular space structure, while the DGCNN adopts the idea of a map to realize a method for deep learning of the data of the irregular space structure; the dynamic graph convolutional neural network algorithm can realize end-to-end learning, and complex features are automatically learned from the original electroencephalogram signals; the width learning system is an incremental learning system without a depth structure, can be used as a classifier for classifying and identifying emotion, and has good effect and very high speed; therefore, the algorithm of deep learning and width learning is more accurate in assessing mental states;
5. the method of the invention can not only adopt the loudspeaker and the in-ear earphone to play the audio, but also can adopt the bone conduction earphone to play the audio; the bone conduction earphone converts sound into mechanical vibration signals with different frequencies, and the mechanical vibration signals are transmitted through skull, organism and nerve tissues of a person; bone conduction eliminates many of the steps of sound wave transmission compared to the traditional sound conduction mode of generating sound waves through a diaphragm; the sound reproduction device can realize clear sound reproduction in a noisy environment, has better audio playing effect compared with the traditional in-ear earphone and loudspeaker, and the sound wave cannot influence others due to diffusion in the air;
6. according to the invention, the mental state evaluation module sends the acquired physiological signals and the acquired mental state evaluation results to the cloud platform, so that the data storage, processing and data analysis operation of the cloud platform are conveniently realized, and the model correction is conveniently carried out.
Drawings
FIG. 1 is a system block diagram of a head wearable device of the present invention;
FIG. 2 is a schematic structural view of the head wearable device of the present invention;
FIG. 3 is a block diagram of an electroencephalogram signal acquisition module in the head wearable apparatus of the present invention;
FIG. 4 is a block diagram of a mental state estimation module in the head wearable device of the present invention;
FIG. 5 is a block diagram of an audio playback module in the head wearable device of the present invention;
wherein 1 is forehead rack, 1.1 is forehead electrode central line brain electricity collection unit, 1.2 is forehead electrode brain electricity collection unit, 2 is locating rack, 2.1 is extension rod body, 2.2 is audio playing seat body, 2.3 is locating part, 3 is scalable adjustment part.
Detailed Description
The invention is described in further detail below with reference to the drawings and the detailed description.
Example 1
The structure of the head wearable device for mental state evaluation and adjustment according to the present embodiment is as shown in fig. 1, including:
an audio playing module for playing the audio source file;
the physiological signal acquisition module is used for acquiring physiological signals of the tested person;
and the mental state evaluation module is used for carrying out emotion recognition and mental state evaluation on the physiological signals acquired by the physiological signal acquisition module to obtain mental state evaluation results and feeding the mental state evaluation results back to the audio playing module to adjust the played audio source file.
The head wearable device can collect physiological signals of the tested person to evaluate the mental state of the tested person, and feed back and adjust the playing audio in real time to adjust the playing content, so that the emotion of the tested person is influenced in real time, and the mental state is adjusted.
The head wearable device may be a smart helmet, a smart cap, a smart headband, etc.; the present embodiment is described with respect to an intelligent headband. As shown in fig. 2, the head wearable device further comprises a wearable mount; the wearable frame body comprises a forehead frame 1 and positioning frames connected to two sides of the forehead frame 1; the positioning frame 2 comprises an extension rod body 2.1 connected with the forehead frame 1 and an audio playing seat body 2.2; the audio playing seat body 2.2 is connected below the extending rod body 2.1 to form a positioning part 2.3 for positioning on human ears; the audio playing module is arranged on the audio playing seat body 2.2; the physiological signal acquisition module is an electroencephalogram signal acquisition module which is arranged on the forehead frame 1; the mental state evaluation module is arranged on the positioning frame 2; the electroencephalogram signal acquisition module, the mental state evaluation module and the audio playing module are connected through signals in sequence.
The mental state evaluation module is arranged on the positioning frame 2, namely, the printed circuit board and the electronic components of the mental state evaluation module are arranged in the extension rod body, or in the audio playing seat body, or in the extension rod body and the audio playing seat body.
According to the head wearable device, the mental state evaluation module and the audio playing module are arranged on the positioning frame 2, so that the head wearable device is light in weight front and heavy in weight back, the forehead frame 1 is attached to the forehead of a human body under the action of gravity, and the electroencephalogram signal acquisition module can effectively acquire the electroencephalogram signal of the human body; the locating frame 2 formed by the extended rod body 2.1 and the audio playing seat body 2.2 is in a T shape, the rear side of the T shape is a locating part 2.3, and the locating frame is erected on ears of a human body, so that the locating frame can be stably and conveniently located and fixed, and meanwhile, has good wearing comfort.
The forehead support 1 has an arc shape. The arc-shaped forehead rack 1 is convenient to be attached to the forehead of a human body and can realize elastic deformation to be matched with the size of the head of the human body. The extension rod body 2.1 is connected with the forehead rack 1 through a telescopic adjusting part 3. The distance between the extension rod body 2.1 and the forehead rack 1 can be adjusted by adjusting the telescopic adjusting part 3 according to the size of the head of the human body, so that the positioning rack 2 is better worn on the ears of the human body, the audio playing module is better positioned, and the comfort of wearing of the human body is further improved.
The electroencephalogram signal acquisition module is used for acquiring human electroencephalogram signals and comprises a frontal pole central line electroencephalogram acquisition unit 1.1 arranged on the inner side of the forehead frame 1 and frontal pole electroencephalogram acquisition units 1.2 which are arranged on the inner side of the forehead frame 1 and are positioned on two sides of the frontal pole central line electroencephalogram acquisition unit 1.1. The invention can effectively collect the brain electrical signals from one lead to multiple leads of the human body. In the embodiment, the frontal electrode electroencephalogram acquisition units 1.2 are two, and three-lead electroencephalogram signals of a human body can be acquired. In practical application, more frontal electrode electroencephalogram acquisition units can be arranged to acquire human body multi-lead electroencephalogram signals.
The working method of the head wearable device of the embodiment is characterized in that: the method comprises the following steps:
s1, an audio playing module plays an audio source file and performs auditory stimulation on a tested person;
s2, a physiological signal acquisition module acquires physiological signals of a tested person; the physiological signals comprise any one or more than two of brain electrical signals, heart rate signals, electromyographic signals, skin electrical signals and voice signals;
s3, preprocessing, feature extraction and emotion recognition are carried out on the physiological signals by the mental state evaluation module, and mental states of the testee are evaluated to obtain mental state evaluation results;
and S4, the mental state evaluation module feeds back a mental state evaluation result to the audio playing module so as to adjust the audio source file.
The method can collect the physiological signals of the testee to evaluate the mental state of the testee, and feed back and adjust the playing audio in real time to adjust the playing content, thereby affecting the emotion of the testee in real time and further adjusting the mental state.
In the step S1, the audio playing module plays the audio source file, and performs auditory stimulation on the tested person, which means that: the audio playing module is a loudspeaker, an in-ear earphone or a bone conduction earphone; the loudspeaker or the in-ear earphone or the bone conduction earphone plays the audio source file to perform auditory stimulus on the tested person. The method of the invention can not only adopt the loudspeaker and the in-ear earphone to play the audio, but also can adopt the bone conduction earphone to play the audio; the bone conduction earphone as shown in fig. 3 converts sound into mechanical vibration signals of different frequencies, and transmits the signals through skull, body and nerve tissues of a human; bone conduction eliminates many of the steps of sound wave transmission compared to the traditional sound conduction mode of generating sound waves through a diaphragm; and can realize clear sound reduction in noisy environment, compare traditional in-ear earphone and speaker have more outstanding audio playback effect, and sound wave can not influence other people because of diffusing in the air yet. When the audio playing module is a bone conduction earphone, one end of the extending rod body 2.1 is connected with the forehead rack 1, the other end of the extending rod body is used for extending to the rear of human ears, and the audio playing seat body 2.2 is used for being attached to the front sides of mastoid processes of the human ears.
In the step S2, the physiological signal acquisition module acquires physiological signals of the tested person, which means that: the physiological signal acquisition module acquires an electroencephalogram signal of a tested person;
in the step S3, the mental state evaluation module performs preprocessing, feature extraction and emotion recognition on the physiological signal, and evaluates the mental state of the tested person to obtain a mental state evaluation result, which means that: as shown in fig. 4, the mental state evaluation module performs preprocessing on the electroencephalogram signals, performs feature extraction and emotion recognition by adopting a dynamic graph convolution neural network algorithm in combination with a width learning system, evaluates the mental state of the tested person, and obtains a mental state evaluation result.
The mental state evaluation module preprocesses the electroencephalogram signals, adopts a dynamic graph convolution neural network algorithm to combine with a width learning system to perform feature extraction and emotion recognition, evaluates the mental state of a tested person, and obtains a mental state evaluation result, which means that: removing the artifacts of electro-oculogram, electrocardiographic and electromagnetic interference in the electroencephalogram by adopting an independent component analysis algorithm and a principal component analysis algorithm so as to realize pretreatment; extracting the characteristics of the electroencephalogram signals by adopting a dynamic graph convolutional neural network algorithm, mapping the electroencephalogram signals to a characteristic space, and then adopting a width learning system as a classifier in the characteristic space to carry out emotion classification and identification on the electroencephalogram signals to obtain the intensity values of all emotion categories; and evaluating the mental state of the tested person according to the intensity value of each emotion type.
In the mental state evaluation module, a dynamic graph convolution neural network algorithm (Dynamical Graph Convolutional Neural Networks, DGCNN) and a width learning system (Broad Learning System, BLS) are adopted, namely, the algorithm of DGCNN+BLS is adopted to realize the evaluation of mental states; the dynamic graph convolution neural network algorithm is an extension of the convolutional neural network algorithm (CNN) on the graph; the research object of the traditional CNN is mainly aimed at the data of the regular space structure, while the DGCNN adopts the idea of a map to realize a method for deep learning of the data of the irregular space structure; the dynamic graph convolutional neural network algorithm can realize end-to-end learning, and complex features are automatically learned from the original electroencephalogram signals; the width learning system is an incremental learning system without a depth structure, can be used as a classifier for classifying and identifying emotion, and has good effect and very high speed; the deep learning + width learning algorithm is therefore more accurate in assessing mental states.
The emotional categories include fatigue, depression, and boring; assessing the mental state of the subject based on the intensity values of the respective emotion categories means: the mental state of the subject is evaluated by setting a fatigue normal value, a depression normal value and a boring normal value, and judging the ratio of the fatigue strength value to the fatigue normal value, the ratio of the depression strength value to the depression normal value and the boring strength value to the boring normal value, respectively.
In the step S4, the mental state evaluation module feeds back the mental state evaluation result to the audio playing module to adjust the audio source file, which means that: as shown in fig. 5, the mental state evaluation module sends the mental state evaluation result to the external terminal device, the external terminal device calls out a corresponding audio source file from a playing audio library in the external terminal device according to the mental state evaluation result, and sends the audio source file to the audio playing module, and the audio playing module plays the received audio source file to adjust playing audio. The external terminal equipment adjusts the audio played by the audio playing module; the external terminal equipment refers to mobile phones, tablet computers, PCs and other equipment.
The preferable scheme is as follows: the head wearable device further comprises a wireless communication module, and the mental state evaluation module is further in signal connection with the cloud platform through the wireless communication module so as to send physiological signals and mental state evaluation results to the cloud platform. According to the invention, the mental state evaluation module sends the acquired physiological signals and the acquired mental state evaluation results to the cloud platform, so that the data storage, processing and data analysis operation of the cloud platform are conveniently realized, and the model correction is conveniently carried out.
The head wearable device can be used in the medical field and used as an auxiliary facility for diagnosis and treatment; can also be used in the field of daily health care products for regulating daily emotion.
Example two
The working method of the head wearable device of the present embodiment differs from that of the first embodiment in that: in this embodiment, in step S3, the mental state evaluation module performs preprocessing, feature extraction and emotion recognition on the physiological signal, and evaluates the mental state of the tested person to obtain a mental state evaluation result, which refers to using the prior art, such as wavelet analysis and bispectrum analysis. The remaining steps of the working method of this embodiment are the same as those of the first embodiment.
Example III
The working method of the head wearable device of the present embodiment differs from that of the first embodiment in that: in this embodiment, in step S4, the mental state estimation module feeds back the mental state estimation result to the audio playing module to adjust the audio source file, which means: the mental state evaluation module calls out a corresponding audio source file from a play audio library in the mental state evaluation module according to the mental state evaluation result, and sends the audio source file to the audio play module, and the audio play module plays the received audio source file to adjust play audio;
or the mental state evaluation module generates a corresponding audio signal generation instruction according to the mental state evaluation result and sends the audio signal generation instruction to the audio playing module; the audio playing module calls out the corresponding audio source file from the playing audio library in the audio playing module according to the audio signal instruction to play so as to adjust the playing audio. The remaining steps of the working method of this embodiment are the same as those of the first embodiment.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (7)
1. A head wearable device for mental state assessment and adjustment, characterized by: comprising the following steps:
an audio playing module for playing the audio source file;
the physiological signal acquisition module is used for acquiring physiological signals of the tested person;
the mental state evaluation module is used for carrying out emotion recognition and mental state evaluation on the physiological signals acquired by the physiological signal acquisition module to obtain mental state evaluation results and feeding the mental state evaluation results back to the audio playing module to adjust the played audio source file;
the working method of the head wearable device for mental state evaluation and adjustment comprises the following steps:
s1, an audio playing module plays an audio source file and performs auditory stimulation on a tested person;
s2, a physiological signal acquisition module acquires physiological signals of a tested person; the physiological signal acquisition module acquires an electroencephalogram signal of a tested person;
s3, preprocessing, feature extraction and emotion recognition are carried out on the physiological signals by the mental state evaluation module, and mental states of the testee are evaluated to obtain mental state evaluation results;
removing the artifacts of electro-oculogram, electrocardiographic and electromagnetic interference in the electroencephalogram by adopting an independent component analysis algorithm and a principal component analysis algorithm so as to realize pretreatment; extracting the characteristics of the electroencephalogram signals by adopting a dynamic graph convolutional neural network algorithm, mapping the electroencephalogram signals to a characteristic space, and then adopting a width learning system as a classifier in the characteristic space to carry out emotion classification and identification on the electroencephalogram signals to obtain the intensity values of all emotion categories; according to the intensity values of the emotion categories, the mental state of the testee is estimated, and a mental state estimation result is obtained;
and S4, the mental state evaluation module feeds back a mental state evaluation result to the audio playing module so as to adjust the audio source file.
2. The head wearable device for mental state assessment and adjustment according to claim 1, wherein: the device also comprises a wearable frame body; the wearable frame body comprises a forehead frame and positioning frames connected to two sides of the forehead frame; the positioning frame comprises an extension rod body connected with the forehead frame and an audio playing seat body connected below the extension rod body; the audio playing seat body and the extension rod body form a positioning part for positioning on human ears; the audio playing module is arranged on the audio playing seat body; the physiological signal acquisition module is an electroencephalogram signal acquisition module; the electroencephalogram signal acquisition module is arranged on the forehead frame; the mental state evaluation module is arranged on the positioning frame; the electroencephalogram signal acquisition module, the mental state evaluation module and the audio playing module are sequentially connected through signals.
3. The head wearable device for mental state assessment and adjustment according to claim 2, wherein: the electroencephalogram signal acquisition module comprises a frontal pole central line electroencephalogram acquisition unit arranged on the inner side of the forehead frame and frontal pole electroencephalogram acquisition units which are arranged on the inner side of the forehead frame and are positioned on two sides of the frontal pole central line electroencephalogram acquisition unit.
4. The head wearable device for mental state assessment and adjustment according to claim 1, wherein: in the step S1, the audio playing module plays the audio source file, and performs auditory stimulation on the tested person, which means that: the audio playing module is a loudspeaker, an in-ear earphone or a bone conduction earphone; the loudspeaker or the in-ear earphone or the bone conduction earphone plays the audio source file to perform auditory stimulus on the tested person.
5. The head wearable device for mental state assessment and adjustment according to claim 1, wherein: the emotional categories include fatigue, depression, and boring; assessing the mental state of the subject based on the intensity values of the respective emotion categories means: the mental state of the subject is evaluated by setting a fatigue normal value, a depression normal value and a boring normal value, and judging the ratio of the fatigue strength value to the fatigue normal value, the ratio of the depression strength value to the depression normal value and the boring strength value to the boring normal value, respectively.
6. The head wearable device for mental state assessment and adjustment according to claim 1, wherein: in the step S4, the mental state evaluation module feeds back the mental state evaluation result to the audio playing module to adjust the audio source file, which means that: one of three schemes is adopted:
1. the mental state evaluation module calls out a corresponding audio source file from a play audio library in the mental state evaluation module according to the mental state evaluation result, and sends the audio source file to the audio play module, and the audio play module plays the received audio source file to adjust play audio;
2. the mental state evaluation module generates a corresponding audio signal generation instruction according to the mental state evaluation result and sends the audio signal generation instruction to the audio playing module; the audio playing module calls out a corresponding audio source file from a playing audio library in the audio playing module according to the audio signal instruction to play so as to adjust the playing audio;
3. the mental state evaluation module sends the mental state evaluation result to the external terminal equipment, the external terminal equipment calls out a corresponding audio source file from a playing audio library in the external terminal equipment according to the mental state evaluation result, and sends the audio source file to the audio playing module, and the audio playing module plays the received audio source file to adjust playing audio.
7. The head wearable device for mental state assessment and adjustment according to claim 1, wherein: in the step S4, the mental state evaluation module is further in signal connection with the cloud platform through the wireless communication module, so as to send the physiological signal and the mental state evaluation result to the cloud platform.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811430882.XA CN109350051B (en) | 2018-11-28 | 2018-11-28 | Head wearable device for mental state assessment and adjustment and working method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811430882.XA CN109350051B (en) | 2018-11-28 | 2018-11-28 | Head wearable device for mental state assessment and adjustment and working method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109350051A CN109350051A (en) | 2019-02-19 |
| CN109350051B true CN109350051B (en) | 2023-12-29 |
Family
ID=65343322
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811430882.XA Active CN109350051B (en) | 2018-11-28 | 2018-11-28 | Head wearable device for mental state assessment and adjustment and working method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109350051B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348299B (en) * | 2019-06-04 | 2022-06-10 | 上海交通大学 | Method for recognizing three-dimensional object |
| CN110464344A (en) * | 2019-08-16 | 2019-11-19 | 兰州大学 | The method for collecting the device of eeg signal acquisition and music and its playing music |
| CN112515680B (en) * | 2019-09-19 | 2023-03-31 | 中国科学院半导体研究所 | Wearable brain electrical fatigue monitoring system |
| CN110811648A (en) * | 2019-11-20 | 2020-02-21 | 阿呆科技（北京）有限公司 | Depression tendency evaluation system based on residual convolutional neural network |
| CN110947076B (en) * | 2019-11-27 | 2021-07-16 | 华南理工大学 | Intelligent brain wave music wearable device capable of adjusting mental state |
| CN110947075A (en) * | 2019-11-27 | 2020-04-03 | 华南理工大学 | Personalized mental state adjusting system and method based on brainwave music |
| CN111358475A (en) * | 2020-03-16 | 2020-07-03 | 国网安徽省电力有限公司 | Mental state assessment method and system for high-risk operating personnel in power system |
| CN112596273A (en) * | 2020-12-30 | 2021-04-02 | 华南理工大学 | Intelligent color-changing glasses with electroencephalogram emotion assessment and adjustment functions and control method |
| CN114052734B (en) * | 2021-11-24 | 2022-11-01 | 西安电子科技大学 | Electroencephalogram emotion recognition method based on progressive graph convolution neural network |
| CN115363585B (en) * | 2022-09-04 | 2023-05-23 | 北京中科心研科技有限公司 | Standardized group depression risk screening system and method based on habit removal and film watching tasks |
| CN116671881A (en) * | 2023-08-03 | 2023-09-01 | 北京九叁有方物联网科技有限公司 | Head-wearing brain body operation capability assessment device and method based on graph neural network |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104463218A (en) * | 2014-12-16 | 2015-03-25 | 重庆邮电大学 | sEMG self-adaptive mode recognition method based on on-line SVM and application of method on intelligent wheelchair |
| CN205029825U (en) * | 2015-08-10 | 2016-02-10 | 湖北工业大学 | Intelligence accuse sleep earphone |
| CN205051851U (en) * | 2015-10-30 | 2016-02-24 | 深圳创达云睿智能科技有限公司 | Intelligent earphone of brain electricity |
| CN105893780A (en) * | 2016-05-10 | 2016-08-24 | 华南理工大学 | Mental and psychological assessment method based on VR interaction and brain wave and cerebral blood flow monitoring |
| CN106267514A (en) * | 2016-10-19 | 2017-01-04 | 上海大学 | Feeling control system based on brain electricity feedback |
| WO2017069644A2 (en) * | 2015-10-22 | 2017-04-27 | MBRAINTRAIN LLC Belgrade | Wireless eeg headphones for cognitive tracking and neurofeedback |
| CN106620990A (en) * | 2016-11-24 | 2017-05-10 | 深圳创达云睿智能科技有限公司 | Method and device for monitoring mood |
| CN106725532A (en) * | 2016-12-13 | 2017-05-31 | 兰州大学 | Depression automatic evaluation system and method based on phonetic feature and machine learning |
| CN107595302A (en) * | 2017-09-06 | 2018-01-19 | 西安交通大学 | A kind of device and method that user's state of mind is monitored using EEG signals |
| WO2018129211A1 (en) * | 2017-01-04 | 2018-07-12 | StoryUp, Inc. | System and method for modifying biometric activity using virtual reality therapy |
| CN108363978A (en) * | 2018-02-12 | 2018-08-03 | 华南理工大学 | Using the emotion perception method based on body language of deep learning and UKF |
| CN108461126A (en) * | 2018-03-19 | 2018-08-28 | 傅笑 | In conjunction with virtual reality（VR）The novel intelligent psychological assessment of technology and interfering system |
| CN108564011A (en) * | 2017-08-01 | 2018-09-21 | 南京邮电大学 | A kind of personal identification method that normal form being presented based on brain electricity Rapid Speech |
| CN108888281A (en) * | 2018-08-16 | 2018-11-27 | 华南理工大学 | State of mind appraisal procedure, equipment and system |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8582807B2 (en) * | 2010-03-15 | 2013-11-12 | Nec Laboratories America, Inc. | Systems and methods for determining personal characteristics |
| US10074024B2 (en) * | 2010-06-07 | 2018-09-11 | Affectiva, Inc. | Mental state analysis using blink rate for vehicles |
| EP3212073A4 (en) * | 2014-11-02 | 2018-05-16 | Ngoggle Inc. | Smart audio headphone system |
| US20170352283A1 (en) * | 2016-06-07 | 2017-12-07 | Cerekinetic, Inc. | Self-administered evaluation and training method to improve mental state |
-
2018
- 2018-11-28 CN CN201811430882.XA patent/CN109350051B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104463218A (en) * | 2014-12-16 | 2015-03-25 | 重庆邮电大学 | sEMG self-adaptive mode recognition method based on on-line SVM and application of method on intelligent wheelchair |
| CN205029825U (en) * | 2015-08-10 | 2016-02-10 | 湖北工业大学 | Intelligence accuse sleep earphone |
| WO2017069644A2 (en) * | 2015-10-22 | 2017-04-27 | MBRAINTRAIN LLC Belgrade | Wireless eeg headphones for cognitive tracking and neurofeedback |
| CN205051851U (en) * | 2015-10-30 | 2016-02-24 | 深圳创达云睿智能科技有限公司 | Intelligent earphone of brain electricity |
| CN105893780A (en) * | 2016-05-10 | 2016-08-24 | 华南理工大学 | Mental and psychological assessment method based on VR interaction and brain wave and cerebral blood flow monitoring |
| CN106267514A (en) * | 2016-10-19 | 2017-01-04 | 上海大学 | Feeling control system based on brain electricity feedback |
| CN106620990A (en) * | 2016-11-24 | 2017-05-10 | 深圳创达云睿智能科技有限公司 | Method and device for monitoring mood |
| CN106725532A (en) * | 2016-12-13 | 2017-05-31 | 兰州大学 | Depression automatic evaluation system and method based on phonetic feature and machine learning |
| WO2018129211A1 (en) * | 2017-01-04 | 2018-07-12 | StoryUp, Inc. | System and method for modifying biometric activity using virtual reality therapy |
| CN108564011A (en) * | 2017-08-01 | 2018-09-21 | 南京邮电大学 | A kind of personal identification method that normal form being presented based on brain electricity Rapid Speech |
| CN107595302A (en) * | 2017-09-06 | 2018-01-19 | 西安交通大学 | A kind of device and method that user's state of mind is monitored using EEG signals |
| CN108363978A (en) * | 2018-02-12 | 2018-08-03 | 华南理工大学 | Using the emotion perception method based on body language of deep learning and UKF |
| CN108461126A (en) * | 2018-03-19 | 2018-08-28 | 傅笑 | In conjunction with virtual reality（VR）The novel intelligent psychological assessment of technology and interfering system |
| CN108888281A (en) * | 2018-08-16 | 2018-11-27 | 华南理工大学 | State of mind appraisal procedure, equipment and system |
Non-Patent Citations (3)
| Title |
|---|
| Broad Learning System: a new learning paradigm and system without going deep;C. L. Philip Chen et.al;2017 32nd Youth Academic Annual Conference of Chinese Association of Automation (YAC);全文 * |
| EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks;Tengfei Song et.al;IEEE Transactions on Affective Computing;532 - 541 * |
| 脑电信号与个人情绪状态关联性分析研究;陈萌;李幼军;刘岩;;计算机科学与探索(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109350051A (en) | 2019-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109350051B (en) | Head wearable device for mental state assessment and adjustment and working method thereof | |
| US11185257B2 (en) | Hearing assistance device with brain computer interface | |
| CN109620257B (en) | Mental state intervention and regulation system based on biofeedback and working method thereof | |
| US10178952B2 (en) | Bi-hemispheric brain wave system and method of performing bi-hemispherical brain wave measurements | |
| CN109585021B (en) | Mental state evaluation method based on holographic projection technology | |
| US20230199413A1 (en) | Multimodal hearing assistance devices and systems | |
| WO2020187109A1 (en) | User sleep detection method and system | |
| US11297448B2 (en) | Portable system for gathering and processing data from EEG, EOG, and/or imaging sensors | |
| US20150065813A1 (en) | System for recording and processing signal for diagnosing auditory system and method for recording and processing signal for diagnosing auditory system | |
| US20220218941A1 (en) | A Wearable System for Behind-The-Ear Sensing and Stimulation | |
| US11856355B2 (en) | Information processing system and non-transitory computer readable medium storing program | |
| CN108451505A (en) | The In-Ear sleep stage system of light weight | |
| WO2017031028A1 (en) | Devices, systems and methods for monitoring brain activity and for enabling brain to rebalance | |
| US20220394396A1 (en) | Control of parameters of hearing instrument based on ear canal deformation and concha emg signals | |
| JP7410557B2 (en) | Information processing device and program | |
| CN109284009B (en) | System and method for improving auditory steady-state response brain-computer interface performance | |
| JP2017029323A (en) | Device, terminal, and program for identifying face expression by using myoelectric signal | |
| CN103816007A (en) | Equipment and method for tinnitus treatment based on electroencephalogram frequency-domain characteristic indexation algorithm | |
| CN209932734U (en) | Head wearable equipment | |
| CN114403900A (en) | Electroencephalogram data automatic recording and analyzing system and method in electroencephalogram machine | |
| CN115349871B (en) | Electroencephalogram signal quality real-time judging method and electroencephalogram acquisition device | |
| CN219270939U (en) | Electroencephalogram acquisition device | |
| CN116158770A (en) | Wearable facial surface myoelectricity acquisition device and method | |
| CN114788701A (en) | Eye movement detection and analysis system based on multichannel array electro-ocular electrodes | |
| Paul | In-Ear Electrophysiology for Unobtrusive Auditory Brain-Machine Interfaces |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |