CN114167989B - Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding - Google Patents
Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding Download PDFInfo
- Publication number
- CN114167989B CN114167989B CN202111497041.2A CN202111497041A CN114167989B CN 114167989 B CN114167989 B CN 114167989B CN 202111497041 A CN202111497041 A CN 202111497041A CN 114167989 B CN114167989 B CN 114167989B
- Authority
- CN
- China
- Prior art keywords
- target
- character
- visual
- auditory
- electroencephalogram
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/377—Electroencephalography [EEG] using evoked responses
- A61B5/378—Visual stimuli
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/377—Electroencephalography [EEG] using evoked responses
- A61B5/38—Acoustic or auditory stimuli
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/16—Sound input; Sound output
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Public Health (AREA)
- Biophysics (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Psychology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Psychiatry (AREA)
- Pathology (AREA)
- Veterinary Medicine (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- General Physics & Mathematics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Acoustics & Sound (AREA)
- Dermatology (AREA)
- Neurology (AREA)
- Neurosurgery (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention relates to a brain-controlled spelling method and a brain-controlled spelling system based on visual and auditory inducement and stable decoding, wherein the method comprises the following steps: based on an experimental paradigm of visual-auditory cooperative induction, a two-stage code division multiple access coding mode is adopted to code the target characters and generate visual-auditory stimulation; sending visual and auditory stimuli to a user, and inducing the user to generate a target signal for a target character to be spelled; collecting an electroencephalogram signal generated by a subject; preprocessing the electroencephalogram signals, selecting electroencephalogram signal segments with set lengths to input a domain adaptive fusion discriminant analysis model, decoding the target signals to obtain target characters, and outputting the target characters. The target character to be spelled is coded by adopting a visual and auditory cooperative evoking and two-stage code division multiple access coding mode, so that the strength of the evoked specific electroencephalogram signal is enhanced; and a domain adaptation fusion discriminant analysis recognition model is constructed, the target characters are decoded and output, the character decoding output speed is improved, and the brain-controlled character spelling is more stable and efficient.
Description
Technical Field
The invention relates to the technical field of human-computer interaction and biomedical engineering information service, in particular to a brain-controlled spelling method and a brain-controlled spelling system based on visual and auditory inducement and stable decoding.
Background
Direct information interaction and control channels are established by means of non-invasive electroencephalogram signals, interaction experience of people with speech or movement dysfunction and certain specific limited applications can be greatly improved, and the method can be widely applied to the fields of medical rehabilitation, game entertainment, smart home, military and the like.
The visual sense and auditory sense is used as a main sense channel for obtaining information by human beings, the ratio of the visual sense and auditory sense evoked specific electroencephalogram components in the prior art for brain-controlled spelling is the largest in the information obtained by various senses, the current visual sense and auditory sense evoked specific electroencephalogram components mainly comprise Event Related Potentials (ERP), steady State Visual Evoked Potentials (SSVEP), motion visual evoked potentials (mVEP) and the like, the visual sense and auditory sense evoked ERP cannot synchronously encode characters, one character is encoded, and all character matrixes need to be traversed, so that the time required for outputting one target is longer; SSVEP coding can synchronously code characters, has higher efficiency of outputting the characters, but is easy to cause visual fatigue of users after long-term use; the mVEP can improve the above-mentioned deficiencies to some extent, but the target encoding output rate is still to be improved compared to the SSVEP.
Disclosure of Invention
The invention aims to provide a brain-controlled spelling method and a brain-controlled spelling system based on visual and auditory inducement and stable decoding, which can improve the coding efficiency of target characters.
In order to achieve the purpose, the invention provides the following scheme:
the invention provides a brain-controlled spelling method based on visual and auditory inducement and stable decoding, which comprises the following steps:
based on an experimental paradigm of visual-auditory cooperative induction, a two-stage code division multiple access coding mode is adopted to code the target characters and generate visual-auditory stimulation;
sending visual and auditory stimuli to a user to induce the user to generate a target signal for a target character to be spelled;
collecting an electroencephalogram signal generated by a subject; the electroencephalogram signals comprise target signals and non-target signals;
preprocessing the electroencephalogram signals, selecting the electroencephalogram signal segment input domain with the set length to adapt to the fusion discriminant analysis model, decoding the target signals to obtain target characters, and outputting the target characters.
Optionally, the visual and auditory synergetic evoked experimental paradigm specifically includes:
presenting a visual stimulation interface to a user, and synchronously playing auditory stimulation to the user in sequence; the visual stimulation interface comprises a chunk display interface and a character block display interface; the chunk display interface comprises 8 chunks, each chunk comprises 4 characters arranged according to a square, and a square moving from left to right is arranged at the center of each 4 characters and below each character in each chunk; the character block display interface comprises 1 character block, the character block comprises 4 characters arranged in a square shape, and a square moving from left to right is arranged below each character; the auditory stimulation comprises position information voice sequences which are played circularly in sequence.
Optionally, the encoding the target character by using a two-stage code division multiple access encoding method specifically includes:
by presenting a chunk display interface for 3 times and simultaneously playing a position information voice sequence, when a square block positioned in the center of 4 characters in the chunk display interface containing target characters moves from left to right and the position of the chunk where the target character is positioned in the chunk display interface corresponds to information played in the voice sequence, determining a chunk code of the target character;
by presenting a character block display interface for 2 times and simultaneously playing a position information voice sequence, when a square below a target character moves from left to right in the character block display interface containing the target character and the position of the target character on the character block display interface corresponds to information played in the voice sequence, determining a character code of the target character;
and determining a block code and a character code of a target character according to the 3-time block display interface and the 2-time character block display interface, and completing the two-stage code division multiple access coding of the target character.
Optionally, the method for constructing the domain-adaptive fusion discriminant analysis model specifically includes:
acquiring an electroencephalogram signal segment with a set length of any user from a spelling intention decoding subsystem database, wherein the electroencephalogram signal segment comprises a plurality of electroencephalogram sample data containing classification labels; the classification label comprises a target signal and a non-target signal, the target signal is obtained by inducing the electroencephalogram sample data by a target character, and the non-target class is obtained by inducing the electroencephalogram sample data by a non-target character;
respectively calculating a target sample mean value and a non-target sample mean value according to a plurality of electroencephalogram sample data containing classification labels;
determining an intra-class divergence matrix and an inter-class divergence matrix according to the target class sample mean value and the non-target class sample mean value;
constructing an objective function according to the intra-class divergence matrix, the inter-class divergence matrix and the mapping matrix, and determining an optimal solution of the mapping matrix according to the objective function, the target class sample mean value and the non-target class sample mean value; the mapping matrix is the product of the electroencephalogram sample data and the electroencephalogram sample data mapping matrix;
and determining a domain adaptive fusion discriminant analysis model according to the optimal solution of the mapping matrix, the target sample mean value and the non-target sample mean value.
Optionally, the domain-adaptive fusion discriminant analysis model is determined as follows:
wherein, mu 0 Mean value of non-target class samples, μ 1 Is the target class sample mean, T is the transpose of the matrix,
for the optimal solution of the mapping matrix, ρ represents the difference of the discrimination distances between the target class sample and the non-target class sample, and dist represents the distance.
Optionally, the selecting the electroencephalogram signal segment input domain with the set length to adapt to the fusion discriminant analysis model, and decoding the target signal to obtain the target character specifically includes:
inputting the EEG signal segments with set length into a domain adaptive fusion discriminant analysis model to obtain the difference value of the discriminant distances between the target samples and the non-target samples;
judging whether the difference value is larger than zero, and when the difference value is smaller than zero, judging that the sample data is a non-target signal; and when the difference is larger than zero, judging the sample data as a target signal, and decoding the target signal to output a corresponding target character.
To achieve the above object, the present invention further provides a brain-controlled spelling system based on visual-auditory evoked and stable decoding, the brain-controlled spelling system comprising: the visual and auditory evoked subsystem, the electroencephalogram acquisition subsystem and the spelling intention decoding subsystem;
the visual and auditory evoked sub-system is used for encoding the target character based on an experimental paradigm of visual and auditory cooperative evoked and adopting a two-stage code division multiple access encoding mode, generating visual and auditory stimulation, sending the visual and auditory stimulation to a user, and inducing the user to generate a target signal for the target character to be spelled;
the brain electricity acquisition subsystem is used for acquiring brain electricity signals generated by a testee; the electroencephalogram signals comprise target signals and non-target signals;
and the spelling intention decoding subsystem is used for preprocessing the electroencephalogram signals, selecting the electroencephalogram signal segments with set lengths to input a domain adaptive fusion discriminant analysis model, decoding the target signals to obtain target characters and outputting the target characters.
Optionally, the visual-auditory evoked subsystem comprises:
the character display module is used for presenting a visual stimulation interface to a user;
and the voice playing module is used for playing auditory stimulation to the user.
Optionally, the experimental paradigm of visual-auditory cooperative induction specifically includes:
presenting a visual stimulation interface through a character display module, and synchronously playing auditory stimulation to a user in sequence through a voice playing module; the visual stimulation interface comprises a chunk display interface and a character block display interface; the chunk display interface comprises 8 chunks, each chunk comprises 4 characters arranged in a square, and a square moving from left to right is arranged at the center of each 4 characters and below each character in each chunk; the character block display interface comprises 1 character block, the character block comprises 4 characters which are arranged according to a square shape, and a square block moving from left to right is arranged below each character; the auditory stimulation comprises position information voice sequences which are sequentially played in a circulating way.
Optionally, the encoding the target character by using a two-stage code division multiple access encoding method specifically includes:
by presenting a chunk display interface for 3 times and simultaneously playing a position information voice sequence, when a square block positioned in the center of 4 characters in the chunk display interface containing target characters moves from left to right and the position of the chunk where the target character is positioned in the chunk display interface corresponds to information played in the voice sequence, determining a chunk code of the target character;
by presenting a character block display interface for 2 times and simultaneously playing a position information voice sequence, when a square below a target character moves from left to right in the character block display interface containing the target character and the position of the target character on the character block display interface corresponds to information played in the voice sequence, determining a character code of the target character;
and determining a block code and a character code of a target character according to the 3 times of block display interfaces and the 2 times of character block display interfaces to finish the two-stage code division multiple access coding of the target character.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
the invention provides a brain-controlled spelling method and a brain-controlled spelling system based on visual and auditory inducement and stable decoding, wherein the method comprises the following steps: based on an experimental paradigm of visual-auditory cooperative induction, a two-stage code division multiple access coding mode is adopted to code the target characters and generate visual-auditory stimulation; sending visual and auditory stimuli to a user to induce the user to generate a target signal for a target character to be spelled; collecting an electroencephalogram signal generated by a subject; preprocessing the EEG signals, selecting EEG signal segments with set lengths to input a domain adaptive fusion discriminant analysis model, decoding the target signals to obtain target characters, and outputting the target characters. The target character to be spelled is coded by adopting a visual and auditory cooperative evoking and two-stage code division multiple access coding mode, so that the strength of the evoked specific electroencephalogram signal is enhanced; and a domain adaptation fusion discriminant analysis recognition model is constructed, the target characters are decoded and output, the character decoding output speed is improved, and the brain-controlled character spelling is more stable and efficient.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.
FIG. 1 is a flow chart of a brain-controlled spelling method based on visual-auditory evoked and stable decoding according to the present invention;
FIG. 2 is a block display interface of the present invention;
FIG. 3 is a block diagram of a character display interface according to the present invention;
fig. 4a and 4b are the experimental paradigm of visual and auditory cooperative evoked responses according to the present invention, and the stimulus presentation process is implemented by using a two-level code division multiple access coding method;
FIG. 5 is a schematic diagram of a domain-adaptive fusion discriminant analysis model of the present invention;
FIG. 6 is a schematic diagram of a brain-controlled spelling system based on visual-auditory evoked and stable decoding according to the present invention;
FIG. 7 is a diagram of the difference between the waveforms of the target signal and the non-target signal according to the present invention;
FIG. 8 is a block code and character code based two-stage CDMA coding scheme according to the present invention;
fig. 9 shows the encoding result of two-stage cdma using the character a as an example.
Description of the symbols:
the system comprises a visual and auditory evoked subsystem-1, an electroencephalogram acquisition subsystem-2, an electroencephalogram electrode-21, an electroencephalogram sending module-22, an electroencephalogram receiving module-23 and a spelling intention decoding subsystem-3.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The invention aims to provide a brain-controlled spelling method and a brain-controlled spelling system based on visual and auditory inducement and stable decoding, which can improve the coding efficiency of target characters.
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in further detail below.
As shown in fig. 1, the present invention provides a brain-controlled spelling method based on visual-auditory evoked and stable decoding, which comprises the following steps:
s1: based on the experimental paradigm of visual and auditory cooperative induction, the target character is coded by adopting a two-stage code division multiple access coding mode, and visual and auditory stimuli are generated.
S2: sending visual and auditory stimuli to the user to induce the user to generate a target signal for the target character to be spelled.
S3: collecting an electroencephalogram signal generated by a subject; the electroencephalogram signals include target signals and non-target signals.
S4: preprocessing the electroencephalogram signals, selecting the electroencephalogram signal segment input domain with the set length to adapt to the fusion discriminant analysis model, decoding the target signals to obtain target characters, and outputting the target characters.
Further, the experimental paradigm of visual-auditory cooperative induction specifically includes:
presenting a visual stimulation interface to a user, and synchronously playing auditory stimulation to the user in sequence; the visual stimulation interface comprises a chunk display interface (shown in FIG. 2) and a character chunk display interface (shown in FIG. 3); the chunk display interface comprises 8 chunks, each chunk comprises 4 characters arranged according to a square, and a square moving from left to right is arranged at the center of each 4 characters and below each character in each chunk; the character block display interface comprises 1 character block, the character block comprises 4 characters which are arranged according to a square shape, and a square block moving from left to right is arranged below each character; the auditory stimulation comprises position information voice sequences which are played circularly in sequence.
Further, the encoding the target character by using a two-stage code division multiple access encoding method specifically includes:
by presenting the chunk display interface for 3 times and simultaneously playing the position information voice sequence, when a square block positioned in the center of 4 characters in the chunk display interface containing the target character moves from left to right and the position of the chunk where the target character is positioned in the chunk display interface corresponds to the information played in the voice sequence, the chunk code of the target character is determined.
By presenting the character block display interface for 2 times and simultaneously playing the position information voice sequence, when a square below a target character moves from left to right in the character block display interface containing the target character and the position of the target character on the character block display interface corresponds to the information played in the voice sequence, the character code of the target character is determined.
As shown in fig. 4a and 4b, a block code and a character code of a target character are determined according to the 3-time block display interface and the 2-time character block display interface, and encoding of two-stage code division multiple access of the target character is completed.
Further, the method for constructing the domain-adaptive fusion discriminant analysis model specifically includes:
obtaining EEG signal X of any user from spelling intention decoding subsystem database i ∈R Ns×(Nc×Nt) I =0,1, where i is a category, the corresponding tag information Y belongs to {0,1},0 is a non-target class, 1 is a target class, ns is the electroencephalogram signal sample number (i.e., the number of segments of the acquired electroencephalogram signal), nc =10 is the number of electroencephalogram electrodes, and Nt is the number of electroencephalogram sample points. In the embodiment of the present invention, when the electroencephalogram sampling rate is 100Hz, nt =50. Selecting a segment of an electroencephalogram signal with a set length, the brainThe electric signal segment comprises a plurality of electroencephalogram sample data containing classification labels, the classification labels comprise target signals and non-target signals, the target signals are obtained by inducing the electroencephalogram sample data through target characters, and the non-target types are obtained by inducing the electroencephalogram sample data through the non-target characters.
And respectively calculating the mean value of the target type sample and the mean value of the non-target type sample according to the plurality of electroencephalogram sample data containing the classification labels.
Determining an intra-class divergence matrix and an inter-class divergence matrix according to the target class sample mean value and the non-target class sample mean value; the formula for calculating the intra-class divergence matrix is as follows:
S w ＝∑(x-μ 0 )(x-μ 0 ) T +∑(x-μ 1 )(x-μ 1 ) T
the formula for calculating the inter-class divergence matrix is as follows: (ii) a (ii) a (ii) a
S b ＝(μ 0 -μ 1 )(μ 0 -μ 1 ) T ；
In order to accurately decode the electroencephalogram signal with the spelling intention, so that the intra-class difference is as small as possible, and the inter-class difference is as large as possible, constructing an objective function according to the intra-class divergence matrix, the inter-class divergence matrix and the mapping matrix, and determining the optimal solution of the mapping matrix according to the objective function, the target sample mean value and the non-target sample mean value; the mapping matrix is the product of the electroencephalogram sample data and the electroencephalogram sample data mapping matrix; the formula for constructing the objective function is as follows:
the optimal solution of the mapping matrix is obtained as follows:
determining a domain adaptation fusion discriminant analysis model according to the optimal solution of the mapping matrix, the target sample mean and the non-target sample mean, wherein the domain adaptation fusion discriminant analysis model is as follows:
wherein S is w Is an intra-class divergence matrix, mu 0 Mean value of non-target class samples, μ 1 Is the target class sample mean, T is the transpose of the matrix, S b Is an inter-class divergence matrix, w is a sample mapping matrix, J (w) is an objective function,
for the optimal solution of the mapping matrix, S w -1 And the matrix represents an inverse matrix of the divergence matrix in the class, rho represents the difference value of the discrimination distance between the target class sample and the non-target class sample, and dist represents the distance.
Further, the selecting the electroencephalogram signal segment input domain with the set length to adapt to the fusion discriminant analysis model, and decoding the target signal to obtain the target character specifically includes:
and inputting the electroencephalogram signal segments with set lengths into a domain adaptation fusion discriminant analysis model to obtain a difference value rho of discriminant distances between the target samples and the non-target samples.
Judging whether the difference rho is larger than zero, and when the difference rho is smaller than zero, judging that the sample data is a non-target signal; and when the difference rho is larger than zero, judging the sample data as a target signal, and decoding the target signal to output a corresponding target character.
Further, as shown in fig. 5, since the cognitive responses and the reaction times of different users are different, the generated specific electroencephalogram components have differences in time and peak values, and in order to reduce the number of decoding and checking times of new users, the acquired electroencephalogram signals of N users are selected to respectively solve the mapping matrix w i And calculating the discrimination distance rho i By calculating
And constructing a domain adaptive fusion discriminant analysis model, wherein the model output value is greater than 0 and is a target class, and the model output value is less than 0 and is a non-target class. Specifically, in the embodiment of the present invention, by inputting five times (i.e., a stimulation signal corresponding to a block code + a stimulation signal corresponding to a character code 2 times) of sample data, it is possible to decode whether a segment of an electroencephalogram signal is a target signal, and then output a corresponding target character.
To achieve the above object, as shown in fig. 6, the present invention further provides a brain-controlled spelling system based on visual-auditory evoked and stable decoding, the brain-controlled spelling system comprising: a visual and auditory evoked subsystem 1, an electroencephalogram acquisition subsystem 2 and a spelling intention decoding subsystem 3.
And the visual and auditory evoked subsystem 1 is used for coding the target characters by adopting a two-stage code division multiple access coding mode based on an experimental paradigm of visual and auditory cooperative evoked, generating visual and auditory stimuli, sending the visual and auditory stimuli to a user, and inducing the user to generate a target signal for the target characters to be spelled.
The brain electricity collection subsystem 2 is used for collecting brain electricity signals generated by a testee; the electroencephalogram signals include target signals and non-target signals. And the spelling intention decoding subsystem 3 is used for preprocessing the electroencephalogram signals, selecting the electroencephalogram signal segments with set lengths to input a domain adaptive fusion discriminant analysis model, decoding the target signals to obtain target characters and outputting the target characters.
Specifically, the visual-auditory evoked subsystem 1 includes:
and the character display module is used for presenting a visual stimulation interface to a user.
And the voice playing module is used for playing auditory stimulation to the user.
Further, the experimental paradigm of visual-auditory cooperative induction specifically includes:
presenting a visual stimulation interface through a character display module, and synchronously playing auditory stimulation to a user in sequence through a voice playing module; the visual stimulation interface comprises a chunk display interface and a character block display interface; the chunk display interface comprises 8 chunks, each chunk comprises 4 characters arranged in a square, and a square moving from left to right is arranged at the center of each 4 characters and below each character in each chunk; the character block display interface comprises 1 character block, the character block comprises 4 characters which are arranged according to a square shape, and a square block which moves rapidly from left to right is arranged below each character; the auditory stimulation comprises position information voice sequences which are sequentially played in a circulating way.
Further, the encoding the target character by using a two-stage code division multiple access encoding method specifically includes:
by presenting the chunk display interface for 3 times and simultaneously playing the position information voice sequence, when a square block positioned in the center of 4 characters in the chunk display interface containing the target character moves rapidly from left to right and the position of the chunk where the target character is positioned in the chunk display interface corresponds to the information played in the voice sequence, the chunk code of the target character is determined.
By presenting the character block display interface for 2 times and simultaneously playing the position information voice sequence, when a square below a target character in the character block display interface containing the target character moves rapidly from left to right and the position of the target character in the character block display interface corresponds to the information played in the voice sequence, the character code of the target character is determined.
And determining a block code and a character code of a target character according to the 3 times of block display interfaces and the 2 times of character block display interfaces to finish the two-stage code division multiple access coding of the target character.
In the specific embodiment of the invention, the character block display interface comprises 26 English characters of A-Z and 6 functional characters of deletion, carriage return, comma, period, blank and exit, in order to obtain the exercise visual evoked potential, a square block which moves rapidly from left to right is arranged at the center of each group and below each character, and when a person watches a rapidly moving object, a specific electroencephalogram component, namely the exercise visual evoked potential, can be generated. The target character to be spelled adopts the motion vision evoked potential and the voice evoked event related potential related to the position information to carry out two-stage code division multiple access coding of block codes and character codes, a user feels visual and auditory stimuli to induce and generate a specific electroencephalogram signal (namely a target signal), and a time sequence synchronous marking signal is sent to the electroencephalogram acquisition subsystem.
Further, as shown in fig. 6, the electroencephalogram acquisition subsystem 2 includes an electroencephalogram electrode 21, an electroencephalogram transmission module 22 and an electroencephalogram reception module 23, wherein 10 Ag/AgCl electroencephalogram electrodes are placed at positions of TP7, P3, PO7, PO3, TP8, P4, PO8 and PO4 of temporal lobe, parietal lobe and occipital lobe areas of a user's scalp according to the international 10-20 arrangement standard, electroencephalogram signals of the user are acquired in real time and are wirelessly transmitted to the electroencephalogram reception module 23 through the electroencephalogram transmission module 22, and the electroencephalogram reception module 23 is electrically connected with the spelling intention decoding subsystem 3.
Specifically, firstly, storing acquired electroencephalogram and time sequence synchronous marking signals into a spelling intention decoding subsystem database, then preprocessing and denoising the electroencephalogram signals, including common average reference electrode conversion, 0.5-40Hz band-pass filtering, removing ocular and electromyogram artifacts by a typical correlation analysis method, selecting 0 ms-500 ms after stimulation starts as effective segments according to the stimulation time sequence synchronous marking signals, carrying out baseline correction by taking-100 ms-0 ms as a reference, and finally constructing a domain adaptation fusion discriminant analysis model by utilizing the electroencephalogram signal segments to carry out target character decoding output.
The invention takes spelling of a character A as an example, and introduces an electroencephalogram evoked stimulus interface and a character coding mode. As shown in fig. 3, the stimulation presentation mode is 2-level presentation, the time of one-time exercise stimulation is 300ms, the current stimulation test has a fast exercise square marked as 1, and no fast exercise square marked as 0, then i-times fast exercise stimulation can be marked with 2 at the same time i And outputting the target.
The user's intention to select the target character is identified in 2 levels, i.e., the block code is identified first, and then the character code is identified. Taking the character a selection as an example, the block code is presented by 3 stimuli: in the 1 st stimulation, the block where the character A is located has a square block moving rapidly from left to right and is accompanied by the voice "left"; in the 2 nd stimulation, the block in which the character A is located has no square block moving rapidly from left to right and is accompanied by the voice "middle"; in the 3 rd stimulation, the block where the character a is located has no fast moving square from left to right, and the corresponding block code is 100 along with the speech "right". After determining the grouping, the character code is presented by 2 stimuli: in the 1 st stimulation, a square moving rapidly from left to right is arranged below the character A and is accompanied by the voice 'left'; in the 2 nd stimulation, a square block which moves rapidly from left to right is not arranged below the character A, and along with the speech of 'right', the corresponding character code is 10, the waveform difference of the electroencephalogram signal is reflected as shown in fig. 6, and the code is 1, and the electroencephalogram waveform corresponding to the target stimulation has larger amplitude change and is a target signal; a code of 0 corresponds to no significant amplitude change in the non-target stimulated brain waveform, corresponding to a non-target signal. When the target character to be selected is positioned on the left side, a specific electroencephalogram component is induced by hearing the left voice, and the electroencephalogram component cannot exist by hearing the middle or right voice. The encoding method is shown in fig. 8 and fig. 9. The presentation time of each quick movement stimulus is 300ms, the playing time of each voice is 300ms, one quick movement stimulus is finished, all movement blocks disappear for 100ms, and the voice playing is stopped for 100ms, so that the movement blocks are used as the starting mark of the next stimulus test, and the stimulus presentation program can be written through the psychtolbo.
After the spelling of the characters is started, a user sits right in front of a display, eyes watch a screen, a sliding square block in a stimulation presentation interface is presented according to a group code rule of fig. 2, then the character is presented according to a character code rule, 5 times of square block movement from left to right and 5 groups of position information voice playing processes can mark a character to be spelled, the character to be spelled is marked by the user and output, a target character to be spelled adopts a movement vision evoked potential and a position information related voice evoked event related potential to carry out two-stage code division multiple access coding on the group code and the character code, and a time sequence synchronization marking signal is sent to a receiving module of an electroencephalogram acquisition subsystem, so that the user feels and feels auditory stimulation, and induces generation of specific electroencephalogram signals.
The invention relates to a brain-controlled spelling method and a brain-controlled spelling system based on visual and auditory inducement and stable decoding, which fully utilize a code division multiple access mode to carry out code division multiple access synchronous coding on a group code and a character code of a target character, improve the coding efficiency of an mVEP (multi-vector error vector) on the target character, and adopt voice with consistent position information to induce ERP (Enterprise resource planning), thereby enhancing the strength of induced specific electroencephalogram components.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. For the system disclosed by the embodiment, the description is relatively simple because the system corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
The principles and embodiments of the present invention have been described herein using specific examples, which are provided only to help understand the method and the core concept of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, the specific embodiments and the application range may be changed. In view of the above, the present disclosure should not be construed as limiting the invention.
Claims (4)
1. A brain-controlled spelling method based on visual-auditory evoked and stable decoding, the brain-controlled spelling method comprising:
based on an experimental paradigm of visual-auditory cooperative induction, a two-stage code division multiple access coding mode is adopted to code the target characters and generate visual-auditory stimulation;
sending visual and auditory stimuli to a user to induce the user to generate a target signal for a target character to be spelled;
collecting an electroencephalogram signal generated by a subject; the electroencephalogram signals comprise target signals and non-target signals;
the cognitive response and the reaction time of different users are different, and the generated electroencephalogram signals are different in time and peak value; acquiring electroencephalogram signals of N users, obtaining a mapping matrix, calculating a difference value of discrimination distances of a target signal and a non-target signal, and constructing a domain-adaptive fusion discrimination analysis model;
preprocessing the electroencephalogram signal, selecting the electroencephalogram signal segment input domain with the set length to adapt to the fusion discriminant analysis model, decoding the target signal to obtain a target character and outputting the target character, and specifically comprises the following steps:
inputting the EEG signal segments with set length into a domain adaptive fusion discriminant analysis model to obtain the difference value of the discriminant distances between the target samples and the non-target samples;
judging whether the difference value is larger than zero, and when the difference value is smaller than zero, judging that the sample data is a non-target signal; when the difference value is larger than zero, judging the sample data as a target signal, decoding the target signal and outputting a corresponding target character;
the experimental paradigm for visual and auditory cooperative induction specifically includes:
presenting a visual stimulation interface to a user, and synchronously playing auditory stimulation to the user in sequence; the visual stimulation interface comprises a chunk display interface and a character block display interface; the chunk display interface comprises 8 chunks, each chunk comprises 4 characters arranged according to a square, and a square moving from left to right is arranged at the center of each 4 characters and below each character in each chunk; the character block display interface comprises 1 character block, the character block comprises 4 characters which are arranged according to a square shape, and a square block moving from left to right is arranged below each character; the auditory stimulation comprises a position information voice sequence which is sequentially played in a circulating way;
the encoding of the target character by adopting a two-stage code division multiple access encoding mode specifically comprises the following steps:
by presenting a chunk display interface for 3 times and simultaneously playing a position information voice sequence, when a square block positioned in the center of 4 characters in the chunk display interface containing target characters moves from left to right and the position of the chunk where the target character is positioned in the chunk display interface corresponds to information played in the voice sequence, determining a chunk code of the target character;
by presenting a character block display interface for 2 times and simultaneously playing a position information voice sequence, when a square below a target character moves from left to right in the character block display interface containing the target character and the position of the target character on the character block display interface corresponds to information played in the voice sequence, determining a character code of the target character;
determining a block code and a character code of a target character according to the 3 times of block display interfaces and the 2 times of character block display interfaces to complete the coding of two-stage code division multiple access of the target character;
and decoding whether a section of electroencephalogram signal segment is a target signal or not according to the 3 times of block display interfaces and the 2 times of character block display interfaces, and finishing outputting corresponding target characters.
2. The brain-controlled spelling method based on visual-auditory evoked and stable decoding according to claim 1, wherein the method for constructing the domain-adaptive fusion discriminant analysis model specifically comprises:
acquiring an electroencephalogram signal segment with a set length of any user from a spelling intention decoding subsystem database, wherein the electroencephalogram signal segment comprises a plurality of electroencephalogram sample data containing classification labels; the classification label comprises a target signal and a non-target signal, the target signal is obtained by inducing electroencephalogram sample data by a target character, and the non-target class is obtained by inducing the electroencephalogram sample data by a non-target character;
respectively calculating a target sample mean value and a non-target sample mean value according to a plurality of electroencephalogram sample data containing classification labels;
determining an intra-class divergence matrix and an inter-class divergence matrix according to the target class sample mean value and the non-target class sample mean value;
constructing an objective function according to the intra-class divergence matrix, the inter-class divergence matrix and the mapping matrix, and determining an optimal solution of the mapping matrix according to the objective function, the target class sample mean value and the non-target class sample mean value; the mapping matrix is the product of the electroencephalogram sample data and the electroencephalogram sample data mapping matrix;
and determining a domain adaptive fusion discriminant analysis model according to the optimal solution of the mapping matrix, the target sample mean value and the non-target sample mean value.
3. The brain-controlled spelling method based on visual-auditory evoked and stable decoding as claimed in claim 1, wherein the domain-adapted fusion discriminant analysis model is determined as:
wherein, mu 0 Mean value of non-target class sample, μ 1 Is the target class sample mean, T is the transpose of the matrix,
for the optimal solution of the mapping matrix, ρ represents the difference of the discrimination distances between the target class sample and the non-target class sample, and dist represents the distance.
4. A brain-controlled spelling system based on visual-auditory evoked and stable decoding, the brain-controlled spelling system comprising: the visual and auditory evoked subsystem, the electroencephalogram acquisition subsystem and the spelling intention decoding subsystem;
the visual and auditory evoked subsystem is used for coding the target characters by adopting a two-stage code division multiple access coding mode based on an experimental paradigm of visual and auditory cooperative evoked, generating visual and auditory stimuli, sending the visual and auditory stimuli to a user and inducing the user to generate a target signal for the target characters to be spelled;
the brain electricity acquisition subsystem is used for acquiring brain electricity signals generated by a testee; the electroencephalogram signals comprise target signals and non-target signals;
the cognitive response and the reaction time of different users are different, and the generated electroencephalogram signals are different in time and peak value; acquiring electroencephalogram signals of N users, obtaining a mapping matrix, calculating a difference value of discrimination distances of a target signal and a non-target signal, and constructing a domain-adaptive fusion discrimination analysis model;
the spelling intention decoding subsystem is used for preprocessing the electroencephalogram signals, selecting the electroencephalogram signal segment input domain with the set length to adapt to the fusion discriminant analysis model, decoding the target signals to obtain target characters and outputting the target characters, and specifically comprises the following steps:
inputting the EEG signal segments with set length into a domain adaptive fusion discriminant analysis model to obtain the difference value of the discriminant distances between the target samples and the non-target samples;
judging whether the difference value is larger than zero, and when the difference value is smaller than zero, judging that the sample data is a non-target signal; when the difference value is larger than zero, judging the sample data as a target signal, decoding the target signal and outputting a corresponding target character;
the visual-auditory evoked subsystem comprises:
the character display module is used for presenting a visual stimulation interface to a user;
the voice playing module is used for playing auditory stimulation to a user;
the experimental paradigm for visual and auditory cooperative induction specifically includes:
presenting a visual stimulation interface through a character display module, and synchronously playing auditory stimulation to a user in sequence through a voice playing module; the visual stimulation interface comprises a chunk display interface and a character block display interface; the chunk display interface comprises 8 chunks, each chunk comprises 4 characters arranged in a square, and a square moving from left to right is arranged at the center of each 4 characters and below each character in each chunk; the character block display interface comprises 1 character block, the character block comprises 4 characters which are arranged according to a square shape, and a square block moving from left to right is arranged below each character; the auditory stimulation comprises position information voice sequences which are sequentially played in a circulating way;
the encoding of the target character by adopting a two-stage code division multiple access encoding mode specifically comprises the following steps:
by presenting a chunk display interface for 3 times and simultaneously playing a position information voice sequence, when a square block positioned in the center of 4 characters in the chunk display interface containing target characters moves from left to right and the position of the chunk where the target character is positioned in the chunk display interface corresponds to information played in the voice sequence, determining a chunk code of the target character;
by presenting a character block display interface for 2 times and simultaneously playing a position information voice sequence, when a square below a target character in the character block display interface containing the target character moves from left to right and the position of the target character in the character block display interface corresponds to information played in the voice sequence, determining a character code of the target character;
determining a block code and a character code of a target character according to the 3 times of block display interfaces and the 2 times of character block display interfaces to complete the coding of two-stage code division multiple access of the target character;
and decoding whether a section of electroencephalogram signal segment is a target signal or not according to the 3 times of block display interfaces and the 2 times of character block display interfaces, and finishing outputting a corresponding target character.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111497041.2A CN114167989B (en) | 2021-12-09 | 2021-12-09 | Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111497041.2A CN114167989B (en) | 2021-12-09 | 2021-12-09 | Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114167989A CN114167989A (en) | 2022-03-11 |
| CN114167989B true CN114167989B (en) | 2023-04-07 |
Family
ID=80484781
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111497041.2A Active CN114167989B (en) | 2021-12-09 | 2021-12-09 | Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114167989B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109521870A (en) * | 2018-10-15 | 2019-03-26 | 天津大学 | A kind of brain-computer interface method that the audio visual based on RSVP normal form combines |
| CN110347242A (en) * | 2019-05-29 | 2019-10-18 | 长春理工大学 | Audio visual brain-computer interface spelling system and its method based on space and semantic congruence |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7269456B2 (en) * | 2002-05-30 | 2007-09-11 | Collura Thomas F | Repetitive visual stimulation to EEG neurofeedback protocols |
| CN108469896A (en) * | 2018-01-23 | 2018-08-31 | 天津大学 | A kind of brain-computer interface hybrid coding normal form method based on asymmetric brain electrical feature |
| CN111338482B (en) * | 2020-03-04 | 2023-04-25 | 太原理工大学 | Brain-controlled character spelling recognition method and system based on supervision self-coding |
-
2021
- 2021-12-09 CN CN202111497041.2A patent/CN114167989B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109521870A (en) * | 2018-10-15 | 2019-03-26 | 天津大学 | A kind of brain-computer interface method that the audio visual based on RSVP normal form combines |
| CN110347242A (en) * | 2019-05-29 | 2019-10-18 | 长春理工大学 | Audio visual brain-computer interface spelling system and its method based on space and semantic congruence |
Non-Patent Citations (2)
| Title |
|---|
| Ce Zhang ; Azim Eskandarian."A Survey and Tutorial of EEG-Based Brain Monitoring for Driver State Analysis".《IEEE/CAA Journal of Automatica Sinica》.2021,全文. * |
| 郭柳君 ; 张雪英 ; 陈桂军 ; .基于空-频域特征的视听混合脑机接口.计算机工程与设计.2020,(06),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114167989A (en) | 2022-03-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106569604B (en) | Audiovisual bimodal semantic matches and semantic mismatch collaboration stimulation brain-machine interface method | |
| CN101464729B (en) | Independent desire expression method based on auditory sense cognition neural signal | |
| Gouy-Pailler et al. | Nonstationary brain source separation for multiclass motor imagery | |
| Yin et al. | A speedy hybrid BCI spelling approach combining P300 and SSVEP | |
| CN104978035B (en) | Brain machine interface system and its implementation based on body-sensing electric stimulus inducing P300 | |
| Blankertz et al. | Optimizing spatial filters for robust EEG single-trial analysis | |
| CN101515199B (en) | Character input device based on eye tracking and P300 electrical potential of the brain electricity | |
| CN105824418B (en) | A kind of brain-computer interface communication system based on asymmetric visual evoked potential | |
| CN104965584A (en) | Mixing method for brain-computer interface based on SSVEP and OSP | |
| CN114533086B (en) | Motor imagery brain electrolysis code method based on airspace characteristic time-frequency transformation | |
| CN108721048A (en) | Rehabilitation training control method, computer readable storage medium and terminal | |
| CN102200833A (en) | Speller brain-computer interface (SCI) system and control method thereof | |
| CN109034015A (en) | The demodulating system and demodulating algorithm of FSK-SSVEP | |
| CN117389441B (en) | Writing imagination Chinese character track determining method and system based on visual following assistance | |
| CN106056602B (en) | FMRI visual performance datum target extracting method based on CNN | |
| CN108469896A (en) | A kind of brain-computer interface hybrid coding normal form method based on asymmetric brain electrical feature | |
| CN112732082A (en) | Brain-computer interface method based on additional Chinese character acquiescent reading motor imagery mode | |
| CN114145745B (en) | Graph-based multitasking self-supervision emotion recognition method | |
| CN115659207A (en) | Electroencephalogram emotion recognition method and system | |
| CN114167989B (en) | Brain-controlled spelling method and system based on visual and auditory inducement and stable decoding | |
| Selim et al. | Machine learning methodologies in P300 speller Brain-Computer Interface systems | |
| RU2704497C1 (en) | Method for forming brain-computer control system | |
| CN111338482A (en) | Brain-controlled character spelling recognition method and system based on supervised self-encoding | |
| Xue et al. | Improve P300 speller performance by changing stimulus onset asynchrony (SOA) without retraining the subject-independent model | |
| Cecotti et al. | Suboptimal sensor subset evaluation in a p300 brain-computer interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |