CN112230768B

CN112230768B - Wheelchair driven by SSMVEP-ERP-OSR hybrid brain-computer interface

Info

Publication number: CN112230768B
Application number: CN202011068945.9A
Authority: CN
Inventors: 郭晓辉; 王晶; 石斌; 王璐; 梁文栋; 乌铭远
Original assignee: Shenzhen Rhb Medical Tech Co ltd
Current assignee: Shenzhen Rhb Medical Tech Co ltd
Priority date: 2020-09-30
Filing date: 2020-09-30
Publication date: 2023-05-23
Anticipated expiration: 2040-09-30
Also published as: CN112230768A

Abstract

The invention discloses a wheelchair driven by an SSMVEP-ERP-OSR hybrid brain-computer interface. The wheelchair comprises a wheelchair body, wherein a human-computer interaction interface is arranged on the wheelchair body, and the wheelchair further comprises a visual induction module, a data acquisition module, a data preprocessing module, a data identification judging module and a wheelchair driving module, wherein the wheelchair driving module is in communication connection with the data identification judging module, the wheelchair driving module comprises a driving motor and a wheelchair transmission system, and the wheelchair driving module drives the motor to realize driving operation of the wheelchair according to a target control instruction of the data identification judging module. The target number is increased based on the SSMVEP-ERP-OSR mixed brain-computer interface paradigm, so that more instructions for wheelchair control are improved, the flexibility of a subject in controlling the wheelchair is improved, a time-frequency characteristic step-by-step recognition algorithm based on a complex template and CCA is provided, and the accuracy and instantaneity of control are improved through selection of an optimal channel.

Description

Wheelchair driven by SSMVEP-ERP-OSR hybrid brain-computer interface

Technical Field

The invention relates to the field of electroencephalogram signal recognition control, in particular to a wheelchair driven by a mixed brain-computer interface based on SSMVEP-ERP-OSR.

Background

The number of disabled lower limbs is gradually rising due to frequent occurrence of traffic accidents, industrial injuries, diseases and the like. According to the data obtained by sampling and adjusting the disabled people in 2006 in China, the total number of disabled people in China is 8296 ten thousand, wherein the number of disabled people in limbs is 2412 ten thousand, and the number of disabled people is 29.07%. Most lower limb disabled people utilize the electric wheelchair to travel daily, and people sitting on the wheelchair can realize operations such as start-stop, forward movement, backward movement and the like of the wheelchair by operating a button or an operating lever on the wheelchair, but for old people and disabled people with inconvenient hand movements, the operability of the wheelchair is greatly limited, and the convenience of the wheelchair is reduced to a certain extent. The brain-computer interface is a short name of a human brain-computer interface, and is a technology for realizing direct communication and control between the brain and the electronic equipment based on brain electrical signals. The brain-computer interface opens up a brand-new way for information exchange and control with the outside for the human brain because of not depending on a conventional brain output channel.

Visual evoked potential (Visual Evoked Potential, VEP) is an electrical response of the occipital region of the brain to visual stimuli, and is a potential change that represents the response of the retina to visual stimuli, via the visual pathway, to the occipital cortex. When the stimulation frequency of the visual stimulus is above 6Hz, the response of the brain visual system to the external continuous periodic visual stimulus is the steady-state visual evoked potential (Steady State Visually Evoked Potential, SSVEP). However, in the traditional SSVEP-BCI field, most studies are based on flicker or contrast variation of flicker, with few studies of the effect of motion on visual stimuli and their design of potentially steady-state BCI. Steady state motor vision evoked potential (ssmvp-State Motion Visually Evoked Potentials) is a motion-specific visual stimulus that is similar to the periodic expansion and contraction movements of newton's rings. Motion perception, similar to light perception and color contrast, is also one of the basic tasks for the human visual system. The BCI system based on the steady-state movement visual evoked potential has the main advantages that training is not needed, signal acquisition is easy, and visual fatigue can be reduced compared with a light flicker paradigm. The default stimulus response (Omitted stimulus response, OSR) is an endogenous brain response, which refers to a series of pattern features that can cause scalp potential by cessation of repeated sensory stimulus, and which can be induced after the absence of a series of regular sensory stimulus.

Brain-computer interface technology has been widely introduced into the control technology of intelligent wheelchairs, wherein the brain-computer interface technology mainly comprises a motor imagery and steady-state vision induced brain-computer interface. The intelligent wheelchair system based on the motor imagery brain electrical control mainly has the defects of low accuracy, large individual difference and the like, and the wheelchair system based on the steady-state visual evoked potential control mainly has the defects of less control instructions, long training time and the like.

Disclosure of Invention

The invention aims to provide a wheelchair based on an SSMVEP-ERP-OSR hybrid brain-computer interface drive, which solves at least one of the existing technical problems.

In order to achieve the above purpose, the invention adopts the following technical scheme that the wheelchair driven by the SSMVEP-ERP-OSR hybrid brain-computer interface comprises a wheelchair body, wherein the wheelchair body is provided with a man-machine interaction interface, and the wheelchair further comprises:

visual induction module: the human-computer interaction interface is provided with a visual induction module, the visual induction module is embodied in the manner that the human-computer interaction interface presents SSMVEP-ERP-OSR mixed paradigm stimulation, the SSMVEP-ERP-OSR mixed paradigm played by the human-computer interaction interface comprises three rows and three columns of Newton rings, the diameter of each Newton ring is about 4.8deg, the distances between the first column and the third column of Newton rings and the center of a screen are 9.6deg of a visual angle, and the distances between the first row and the third row of Newton rings and the center of the screen are 6.4deg of the visual angle;

and a data acquisition module: the data acquisition module is used for acquiring the brain signals of the user and comprises a 16 conductive electrode cap used for being worn by the user, wherein the 16 conductive electrode cap is provided with a plurality of electrodes, and the electrodes are led according to the international 10/20 standard;

and a data preprocessing module: the data preprocessing module is in communication connection with the data acquisition module, and comprises a data amplifying module, a filtering module and an analog-to-digital conversion module, wherein the data amplifying module is used for amplifying the acquired electroencephalogram signals, the filtering module is used for filtering the amplified electroencephalogram signals, and the analog-to-digital conversion module is used for converting the filtered electroencephalogram signals into electric signals;

and the data identification judging module is used for: the data identification and judgment module is in communication connection with the data preprocessing module, the data identification and judgment module judges and matches a target control instruction of the wheelchair through signal identification, and the target control instruction of the wheelchair comprises forward movement, backward movement, acceleration forward movement, deceleration forward movement, left steering, right steering, acceleration backward movement, deceleration backward movement and stop, and corresponds to three rows and three columns of Newton rings;

wheelchair drive module: the wheelchair driving module is in communication connection with the data identification judging module, and comprises a driving motor and a wheelchair transmission system, and the wheelchair driving module drives the motor to realize driving operation of the wheelchair according to the target control instruction of the data identification judging module.

In some embodiments, the electrodes used for detection include O1, O2, OZ, PO4, PO8, PO3, PO7, P3, CZ, FZ, FCZ, POZ, PZ, P, CPZ, the reference electrode is located at the left ear lobe and the electrode is Fpz.

In some embodiments, the same stimulation target stimulation frequency of the same column, and the corresponding stimulation frequencies of the three columns of newton rings are 15Hz, 17Hz and 19Hz respectively; the stimulation frequencies of the stimulation targets in the same row are different, and the stimulation missing time is different; the target sequence of the first, second and third stimulation with the stimulation frequency of 15Hz comprises three times of short stimulation deletion and one time of long stimulation deletion, and the stimulation deletion moments of the three targets with the same frequency are different.

In some embodiments, the data amplification module is an amplifier, the sampling frequency of which is 1200Hz.

In some embodiments, the filtering module includes 0.05-100Hz band pass filtering and 45-52Hz band reject filtering.

In some embodiments, a security module is also included, the security module being provided with a distance sensor.

The beneficial effects of the invention are as follows: according to the wheelchair driven by the SSMVEP-ERP-OSR mixed brain-computer interface, the target number is increased through the SSMVEP-ERP-OSR mixed brain-computer interface paradigm, so that more instructions for wheelchair control are improved, the flexibility of a subject in wheelchair control is improved, a time-frequency characteristic step-by-step recognition algorithm based on a complex template and CCA is provided, and the accuracy and the instantaneity of control are improved through selection of an optimal channel.

Drawings

FIG. 1 is a schematic diagram of a system framework of a hybrid brain-computer interface driven wheelchair based on SSMVEP-ERP-OSR in accordance with the present invention;

FIG. 2 is a schematic illustration of the placement of electrodes of the electrode cap of the present invention;

FIG. 3 is a schematic illustration of a hybrid paradigm of the present invention;

FIG. 4 is a schematic illustration of the wheelchair target control command of the present invention;

FIG. 5 is a schematic diagram of a time-frequency characteristic distribution recognition algorithm according to the present invention;

FIG. 6 is a timing diagram of an off-line experimental stimulus of the present invention;

FIG. 7 is a flow chart of the basic target classifier generated by the Leave-One-Out Cross-Validation algorithm of the present invention;

FIG. 8 is a flow chart of recording electrode position selection based on an automatic optimizing algorithm according to the present invention.

Detailed Description

The invention is described in further detail below with reference to the accompanying drawings.

Examples

As shown in FIG. 1, the wheelchair driven by the mixed brain-computer interface based on the SSMVEP-ERP-OSR comprises a human-computer interaction module, a signal acquisition module, a signal processing module and an application control module in a system overall framework diagram. Selecting a Windows flag Surface notebook as a system host, wherein the main function of the Windows flag Surface notebook is to provide a man-machine interaction interface and perform data processing; after the system starts to run, firstly, presenting SSMVEP-ERP-OSR mixed paradigm stimulus on a Surface screen, and carrying out visual induction on a user; the user wears a 16-conductive electrode electroencephalogram cap, acquires an electroencephalogram signal which is induced by vision, transmits the EEG signal to Surface after being amplified by an amplifier, processes the data in real time, comprises preprocessing, frequency domain SSMVEP feature extraction and recognition, time domain ERP and OSR feature extraction and recognition, judges a stimulation target, and visually feeds back the recognition result to the user through a computer screen; meanwhile, the identification result is converted into a corresponding instruction, and the instruction is transmitted to a wheelchair control system through wireless communication between a computer and the intelligent wheelchair to control the wheelchair to finish the staring of the subject in the target direction. The specific steps of the above flow are as follows:

step 1: the brain electricity acquisition module has the electrodes of the 16 conductive electrode caps worn by the subject and is arranged on the wheelchair, the distance between the computer screen and the head is 60-80 cm, all electrodes are arranged according to the international 10/20 standard lead, and as shown in figure 2, the recording electrode distribution is mainly positioned in the pillow area, the top area and the central area. The electrodes include O1, O2, OZ, PO4, PO8, PO3, PO7, P3, CZ, FZ, FCZ, POZ, PZ, P, CPZ. The reference electrode was located on the left ear lobe and the electrode was Fpz, ensuring good head to scalp contact and electrode impedance of less than 5 kiloohms during the experiment.

The method comprises the following steps: the man-machine interaction module mainly comprises a mixed paradigm for visually inducing a subject, as shown in fig. 3 and 4, the specific method is that an SSMVEP-ERP-OSP mixed paradigm program written in advance through Matlab is presented on a computer screen, 9 stimulation targets on the screen respectively correspond to Newton rings 1 representing deceleration forward, newton rings 2 representing forward, newton rings 3 representing acceleration forward, newton rings 4 representing left steering, newton rings 5 representing stop, newton rings 6 representing right steering, newton rings 7 representing deceleration backward, newton rings 8 representing backward, and Newton rings 9 representing acceleration backward. The Newton rings are positioned as shown in FIG. 3a, and each Newton ring in the display screen has a diameter of about 4.8 deg., and the first and third rows of Newton rings are each at a distance of 9.6 deg. from the center of the screen, and the first and third rows of Newton rings are each at a distance of 6.4deg. from the center of the screen. The stimulation target stimulation frequencies of the same column are 15Hz, 17Hz and 19Hz respectively; the stimulus frequency varies between stimulus targets in the same row, and the stimulus absence times vary slightly but are about the same. The stimulation time sequence is shown in fig. 3b, and the stimulation frequency is 15Hz of the target sequence of the first, second and third stimulation, wherein the stimulation comprises three times of short stimulation deletion and one time of long stimulation deletion, and the stimulation deletion moments of the three targets with the same frequency are different.

Step 3: after amplification, filtering and analog-to-digital conversion of an electroencephalogram acquisition instrument, inputting the digitized electroencephalogram signals into a computer, wherein a 16-lead gUSBamp amplifier is used as acquisition hardware for acquiring the electroencephalogram signals, the sampling frequency of the amplifier is 1200Hz, and the hardware filtering comprises band-pass filtering of 0.05-100Hz and band-stop filtering of 48-52 Hz.

Step 4, processing the electroencephalogram signals, as shown in fig. 5, comprising the following steps:

step 4-1, preprocessing the brain electrical signals, and removing direct current components in a time sequence and band-pass filtering at 1-45 Hz;

and 4-2, obtaining a basic target classifier of the optimal channel through an offline experiment in a training stage, wherein a mixed brain-computer interface paradigm of the offline experiment is shown in fig. 6. The training stage comprises the steps of staring at nine tasks for nine stimulus targets respectively, wherein each task comprises two Run, each Run comprises 16 trails, each trail is firstly provided with a target prompt of 500ms, then a stimulus presentation of 3000ms is carried out, a subject stares at the prompted target, and then the subject enters the following trail after screen blacking for 500 ms. After the training phase is finished, training data of 32 trail of each stimulation target can be obtained, and a complex template of each stimulation target is generated as a basic target classifier.

Step 4-3, in the training stage, generating a basic target classifier by using a Leave-One-Out Cross-Validation algorithm, as shown in fig. 7, wherein the specific steps are as follows:

a. in the training sample acquisition stage, the subjects look at the stimulation targets in sequence according to the screen prompts, EEG data when looking at different stimulation targets is obtained, 9 staring targets are obtained, and 32 groups of data are acquired for each staring target. Preprocessing original electroencephalogram data through filtering and the like, and storing the data according to different gaze targets and categories, wherein 9 categories of data exist, and each category of data comprises 32 groups;

b. randomly taking one group of data from each type of data as test data TestData, and taking the rest 31 groups of data as training data;

c. the training data is subjected to 31 times of superposition and average to obtain waveform Template templates of each target, and templates of 9 targets form a classifier;

d. and performing typical correlation analysis on the test data TestData of the 9 targets and templates in the classifier respectively, and if the correlation between a certain group of test data and a certain group of templates in the classifier is maximum, considering that the group of test data and the group of templates correspond to the same stimulation target.

e. And classifying 9 groups of test data, if the test data of 9 targets are correctly classified, namely the classification accuracy is 1, inputting the TEMPLATEs in the classifier at the moment into a final TEMPLATE TEMPLATE, otherwise, discarding the group of TEMPLATEs.

And (c) returning to the step (b), randomly taking a group of data from each type of data as test data TestData, repeating the process (b-e), iterating the process 100 times, and averaging TEMPLATEs in TEMPLATE through superposition to generate a mature target classifier, wherein waveform TEMPLATEs of all targets are stored in the classifier.

Step 4-4, in the training stage, an optimal recording electrode position selection algorithm is adopted to automatically find the optimal measuring electrode position, as shown in fig. 8.

Arranging a plurality of electrodes, performing an off-line experiment, and recording EEG signals;

calculating offline accuracy by selecting one channel each time, and selecting a channel with highest accuracy as a fixed channel C1;

sequentially combining the fixed channels with other channels, selecting C2 with highest accuracy if the accuracy is not increased, wherein the optimal channel is [ C1, C2], selecting C2 with highest accuracy if the accuracy is continuously increased, and combining C1 and C2 for subsequent exploration;

d. and continuing to explore according to the thought, if the accuracy cannot be increased continuously, the fixed channel is the optimal channel. The algorithm for automatically optimizing and selecting the optimal recording electrode can avoid the influence of factors such as electrode damage, poor electrode contact, and the like, which are different from person to person.

And 4-4, obtaining a basic target classifier of the optimal channel through the basic target classifier and the automatic optimizing electrode position.

In the step 4-5, in the online test stage, firstly, a tested person looks at a target Newton ring according to a target direction, then, an electroencephalogram signal on an optimal channel obtained in an offline experiment is collected, trend direction removal and filtering pretreatment are carried out, then, SSMVEP frequency domain feature extraction identification based on EMD and CCA is carried out, the specific method is that the electroencephalogram signal is subjected to empirical mode decomposition to obtain a plurality of eigenmode functions (IMFs), wherein SSMVEP features are mainly distributed in the first three components, a signal is reconstructed through the component where the SSMVEP is located, on the basis, SSMVEP features in the reconstruction of the IMF component after the EMD decomposition are identified through a CCA algorithm, so that the flicker frequency of the object where the tested person looks at can be judged, the same stimulation targets in the same column have the same stimulation frequency, and therefore the column where the user looks at the object can be judged.

And 4-6, after the target column is acquired, performing typical correlation analysis on the electroencephalogram signal x (t) on the optimal channel and a complex template of the target corresponding to the stimulation frequency, and finally judging that the subject stares at the target.

Step 5: after the subjects stare at the targets, the identification results are converted into target control instructions so as to control the wheelchair, each target corresponds to the instructions of wheelchair forward, backward, acceleration forward, deceleration forward, left steering, right steering, acceleration backward, deceleration backward and stopping, different targets are selected according to the self requirements of the subjects, the control instructions after the conversion of the identification results are transmitted to the signal receiving module of the wheelchair controller in a wireless manner through the signal processing module, and the wheelchair driving module drives the wheelchair to make different response actions. The sensing system of the intelligent wheelchair makes a protection action according to the change of the surrounding environment of the wheelchair in the process that a subject controls the wheelchair to operate through the distance sensor, so that the aim of safety protection is fulfilled.

What has been described above is merely some embodiments of the present invention. It will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the spirit of the invention.

Claims

1. Wheelchair based on mixed brain-computer interface drive of SSMVEP-ERP-OSR, including the wheelchair body, be provided with man-machine interaction interface on the wheelchair body, its characterized in that still includes:

visual induction module: the human-computer interaction interface is provided with a visual induction module, the visual induction module is embodied in the manner that SSMVEP-ERP-OSR mixed paradigm stimulation is presented on the human-computer interaction interface, the SSMVEP-ERP-OSR mixed paradigm played by the human-computer interaction interface comprises three rows and three columns of Newton rings, the diameter of each Newton ring is about 4.8deg, the distances between the first column and the third column of Newton rings and the center of a screen are 9.6deg of the visual angle, and the distances between the first row and the third row of Newton rings and the center of the screen are 6.4deg of the visual angle;

and a data acquisition module: the data acquisition module is used for acquiring the brain electrical signals of a user and comprises a 16 conductive electrode cap used for being worn by the user, wherein the 16 conductive electrode cap is provided with a plurality of electrodes, and the electrodes are led according to the international 10/20 standard;

and a data preprocessing module: the data preprocessing module is in communication connection with the data acquisition module, the data preprocessing module comprises a data amplification module, a filtering module and an analog-to-digital conversion module, the data amplification module is used for amplifying the acquired brain electrical signals, the filtering module is used for filtering the amplified brain electrical signals, and the analog-to-digital conversion module is used for converting the filtered brain electrical signals into electrical signals;

and the data identification judging module is used for: the data identification judging module is in communication connection with the data preprocessing module, the data identification judging module judges and matches a target control instruction of the wheelchair through signal identification, and the target control instruction of the wheelchair comprises forward movement, backward movement, acceleration forward movement, deceleration forward movement, left steering, right steering, acceleration backward movement, deceleration backward movement and stop, and corresponds to three rows and three columns of Newton rings;

wheelchair drive module: the wheelchair driving module is in communication connection with the data identification judging module, the wheelchair driving module comprises a driving motor and a wheelchair transmission system, and the wheelchair driving module drives the motor to realize driving operation of the wheelchair according to a target control instruction of the data identification judging module;

the data identification judging module realizes data judgment and target control of the wheelchair by the following method:

obtaining a basic target classifier of the optimal channel through an offline experiment in a training stage;

in the training stage, a basic target classifier is generated by using a Leave-One-Out Cross-Validation algorithm, and the specific steps are as follows:

in the training sample acquisition stage, a subject sequentially stares at the stimulation targets according to the screen prompt to obtain EEG data when staring at different stimulation targets, 9 staring targets are provided, and 32 groups of data are acquired by each staring target;

randomly taking one group of data from each type of data as test data TestData, and taking the rest 31 groups of data as training data;

the training data is subjected to 31 times of superposition and average to obtain waveform Template templates of each target, and templates of 9 targets form a classifier;

performing typical correlation analysis on test data TestData of 9 targets and templates in the classifier respectively, and if the correlation between a certain group of test data and a certain group of templates in the classifier is maximum, considering that the group of test data and the group of templates correspond to the same stimulation target;

classifying 9 groups of test data, if the test data of 9 targets are correctly classified, namely the classification accuracy is 1, inputting the TEMPLATEs in the classifier at the moment into a final TEMPLATE TEMPLATE, otherwise discarding the group of TEMPLATEs;

randomly taking a group of data from each type of data as test data TestData, repeating the process, iterating the process for 100 times, and averaging TEMPLATEs in TEMPLATE by superposition to generate a mature target classifier, wherein waveform TEMPLATEs of all targets are stored in the classifier;

in the training stage, an optimal recording electrode position selection algorithm is adopted to automatically find the optimal measuring electrode position;

continuing to explore according to the thought, if the accuracy cannot be increased continuously, the fixed channel is the optimal channel;

obtaining a basic target classifier of an optimal channel through the basic target classifier and the automatic optimizing electrode position;

in the online test stage, firstly, a tested person stares at a target Newton ring according to a target direction, then, an electroencephalogram signal on an optimal channel obtained in an offline experiment is collected, trend direction removal and filtering pretreatment are carried out, and SSMVEP frequency domain feature extraction and identification based on EMD and CCA are carried out to judge the line where a user stares at the target;

after a target column is acquired, performing typical correlation analysis on an electroencephalogram signal x (t) on the optimal channel and a complex template of a target corresponding to the stimulation frequency, and finally judging that a subject stares at the target;

after the subject stares at the target, the identification result is converted into a target control instruction so as to control the wheelchair.

2. The ssmvp-ERP-OSR hybrid brain-computer interface-driven wheelchair of claim 1, wherein the electrodes for detection comprise O1, O2, OZ, PO4, PO8, PO3, PO7, P3, CZ, FZ, FCZ, POZ, PZ, P4, CPZ, reference electrode is located at left ear lobe and electrode is Fpz.

3. The wheelchair driven by the mixed brain-computer interface based on the SSMVEP-ERP-OSR according to claim 1, wherein the stimulation target stimulation frequencies of the same column are the same, and the stimulation frequencies corresponding to the three columns of Newton rings are 15Hz, 17Hz and 19Hz respectively; the stimulation frequencies of the stimulation targets in the same row are different, and the stimulation missing time is different; the target sequence of the first, second and third stimulation with the stimulation frequency of 15Hz comprises three times of short stimulation deletion and one time of long stimulation deletion, and the stimulation deletion moments of the three targets with the same frequency are different.

4. The ssmvp-ERP-OSR hybrid brain-computer interface-driven wheelchair of claim 1, wherein the data amplification module is an amplifier with a sampling frequency of 1200Hz.

5. The ssmvp-ERP-OSR hybrid brain-computer interface-driven wheelchair of claim 1, wherein the filtering module comprises 0.05-100Hz bandpass filtering and 45-52Hz bandstop filtering.

6. The ssmvp-ERP-OSR hybrid brain-computer interface-driven wheelchair of claim 1, further comprising a safety module provided with a distance sensor.