CN109308118B - Chinese eye writing signal recognition system based on EOG and recognition method thereof - Google Patents
Chinese eye writing signal recognition system based on EOG and recognition method thereof Download PDFInfo
- Publication number
- CN109308118B CN109308118B CN201811025755.1A CN201811025755A CN109308118B CN 109308118 B CN109308118 B CN 109308118B CN 201811025755 A CN201811025755 A CN 201811025755A CN 109308118 B CN109308118 B CN 109308118B
- Authority
- CN
- China
- Prior art keywords
- stroke
- chinese character
- data
- chinese
- eog
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000009826 distribution Methods 0.000 claims abstract description 38
- 238000007781 pre-processing Methods 0.000 claims abstract description 11
- 230000004424 eye movement Effects 0.000 claims description 16
- 238000001514 detection method Methods 0.000 claims description 14
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000004397 blinking Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 3
- 230000001186 cumulative effect Effects 0.000 claims description 3
- 238000002570 electrooculography Methods 0.000 description 44
- 210000001508 eye Anatomy 0.000 description 23
- 238000002474 experimental method Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 210000005252 bulbus oculi Anatomy 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 208000026072 Motor neurone disease Diseases 0.000 description 2
- 206010002026 amyotrophic lateral sclerosis Diseases 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 208000005264 motor neuron disease Diseases 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 208000021642 Muscular disease Diseases 0.000 description 1
- 208000012902 Nervous system disease Diseases 0.000 description 1
- 206010069689 Spinal column injury Diseases 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- DFBKLUNHFCTMDC-PICURKEMSA-N dieldrin Chemical compound C([C@H]1[C@H]2[C@@]3(Cl)C(Cl)=C([C@]([C@H]22)(Cl)C3(Cl)Cl)Cl)[C@H]2[C@@H]2[C@H]1O2 DFBKLUNHFCTMDC-PICURKEMSA-N 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 230000003387 muscular Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004434 saccadic eye movement Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F3/023—Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
- G06F3/0233—Character input methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Dermatology (AREA)
- General Health & Medical Sciences (AREA)
- Neurology (AREA)
- Neurosurgery (AREA)
- Character Discrimination (AREA)
Abstract
The invention discloses a Chinese eye writing signal identification method based on EOG, which comprises the following steps: acquiring Chinese eye writing data based on an electro-oculogram and preprocessing the data; dividing the preprocessed data into two parts, namely template data and Chinese character stroke data; the method comprises the steps that five template stroke segments of horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke segment data and Chinese character stroke segment data are all sent to a DTW-based classifier for stroke recognition, and a recognition result is processed by a softmax algorithm to obtain probability distribution of a recognized stroke segment sequence; after a Chinese character word stock is established and is coded by one-hot, the probability distribution of the obtained stroke segment sequence is matched with the coded Chinese character with the same stroke to obtain the final predicted Chinese character. The recognition system of the Chinese eye writing signal recognition method based on the EOG has the advantages of high recognition accuracy of the Chinese character symbol, strong expansion capability of a Chinese character library and wide application prospect.
Description
Technical Field
The invention relates to the technical field of brain-computer interfaces, in particular to a Chinese eye writing signal identification system based on EOG and an identification method thereof.
Background
In recent years, there have been increasing numbers of people suffering from muscular and nervous system diseases such as Amyotrophic Lateral Sclerosis (ALS), Motor Neuron Disease (MND), vertebral injuries, and the like. They almost lose their ability to control muscles or communicate with others. However, most of them are still able to rotate the eyeball effectively. Thus, eye movement based systems provide an alternative way for these patients to communicate with a caregiver or peripheral device. An eye writing system is a system in which a user can input characters by moving their eyes in accordance with a predetermined eye movement pattern of different characters.
Currently, video and electro-oculogram (EOG) based methods have been used to implement eye writing systems. The video-based method does not require contact with the user's skin during data acquisition, and provides very accurate gaze direction estimation compared to its EOG detection method. Therefore, it becomes one of the most popular ways to perceive user behavior. Of these, Itoh et al developed a system that inputs characters and controls environmental devices by acquiring and detecting staring of severely physically handicapped people. Abe et al propose an eye gaze input system using a computer and a home camera. The biggest challenge facing this approach is the interference of illumination variations. In addition, it requires a high computational load to process the large number of images collected by the optical sensor. Electrooculography (EOG) is a measurement technique used in this work that offers another possibility to record eye movements. Compared to video-based methods, the electronic Eye (EOG) method has the advantages of low cost, light computational weight and less impact under ambient light variations.
Today, some EOG-based eye writing systems have been developed. For example, Lopez et al developed an eye movement based computer writing system. The user can select any one of the 64 available numbers, letters or characters by three types of basic eye movements (i.e., saccade, gaze and blink). Kwang-Ryeol et al propose an eye writing recognition system based on real-time electro-oculogram delineation using EOG. The system matches the input traces with electro-oculogram traces of various input patterns that are trained to recognize the intended symbol input.
The drawbacks of the above system are mainly: the single English character is taken as a research object, and only 26 basic English characters exist in English, so that the design of an eye writing system is relatively easy. In contrast, the number of Chinese characters is about one hundred thousand, and the number of common characters reaches 3500. Obviously, when designing a chinese eye writing system, recognizing a single character is time consuming and does not guarantee accuracy.
Therefore, it is desirable to provide a new method and system for identifying a chinese eye writing signal based on EOG to solve the above problems.
Disclosure of Invention
The invention aims to solve the technical problem of providing a Chinese eye writing signal recognition system based on EOG and a recognition method thereof, which have the advantages of high Chinese character recognition accuracy, stronger Chinese character library expansion capability and wide application prospect.
In order to solve the technical problems, the invention adopts a technical scheme that: the method for identifying the Chinese eye writing signal based on the EOG comprises the following steps of:
s1: acquiring Chinese eye writing data based on an electro-oculogram and preprocessing the data;
s2: dividing the preprocessed data into template data and Chinese character stroke data:
the template data is the template stroke segment data which is obtained by dividing continuous stroke template data into five templates, namely horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke, by adopting a sliding window blink detection method;
the Chinese character stroke data is the preprocessed original multi-lead data and is divided into Chinese character stroke segment data by a sliding window detection blinking method;
s3: the method comprises the steps that five template stroke segments of horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke segment data and Chinese character stroke segment data are all sent to a DTW-based classifier for stroke recognition, and a recognition result is processed by a softmax algorithm to obtain probability distribution of a recognized stroke segment sequence;
s4: and (4) establishing a Chinese character library and using one-hot coding to match the probability distribution of the stroke segment sequence obtained in the step (S3) with the coded Chinese character with the same stroke to obtain the final predicted Chinese character.
In a preferred embodiment of the present invention, the preprocessing in step S1 is to perform band-pass filtering on the raw multi-lead electro-ocular data, wherein the band-pass filtering has a cutoff frequency of 0.1-8 Hz.
In a preferred embodiment of the present invention, in step S2, the method for detecting blinking by sliding window includes the following steps:
s2.1: taking the potential difference between the collecting electrodes HEOR and HEOL as a horizontal EOG signal, taking the difference between the collecting electrodes VEOU and VEOD as a vertical EOG signal, and processing the filtered vertical EOG signal through a sliding window;
s2.2: setting and initializing a dynamic threshold amp, wherein the value range of the dynamic threshold amp is (180+ M) - (250+ M), wherein M is the result of averaging all EOG amplitude values in a sliding window; comparing the amplitude of each EOG sample in the current sliding window with a dynamic threshold amp, if higher than amp, marking the corresponding point in the blink segment as a blink point, otherwise a non-blink point, and then updating the data in the sliding window by moving the sample points one by one;
s2.3: repeating step S2.2 until the sliding window moves to the end of the EOG signal;
s2.4: when the continuous EOG signal is detected, the blink segment is used for dividing the continuous EOG signal into a series of stroke segments, and therefore template stroke segment data and Chinese character stroke segment data are obtained.
In a preferred embodiment of the present invention, the step S3 includes the following steps:
s3.1: suppose there is a Chinese character with a total stroke number of m, siRepresents the ith stroke, tjRepresents the jth template (i is more than or equal to 1 and less than or equal to m, j is more than or equal to 1 and less than or equal to 5) and each stroke(s) of the Chinese characteri) Match 5 template strokes and calculate similarity using DTW method:
p(si，tj)＝1-dtw(si，tj)/max1≤k≤5(dtw(si，tk)) (1)
in equation (1), dtw(s)i，tj) Is the cumulative distance, p(s), calculated by the DTW methodi，tj) Representing strokes siIs classified as a template tjProbability of reflecting Chinese character stroke segment siPen combined with templateDrawing segment tjThe similarity between them;
s3.2: p(s) of all strokes of Chinese characteri，tj) Are combined to obtain a similarity matrix P:
pi＝[p(si，t1)，p(si，t2)，…， (3)
wherein p isiIs the probability distribution of the ith stroke divided into five template strokes;
s3.3: and (3) processing the similarity by adopting a softmax algorithm method, so that the similarity calculated by DTW becomes a probability distribution matrix Q, as shown in formulas (4) and (5):
in equation (5), Q represents the recognition probability distribution matrix of the m strokes of a Chinese character.
In a preferred embodiment of the present invention, the step S4 includes the following steps:
s4.1: establishing a Chinese character font library and coding by one-hot, wherein all Chinese characters are composed of five basic strokes of horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke, and the five types of strokes are expressed by one-hot:
1. cross section [ one ] 10000
2. 01000 in vertical axis
3. (iii) 00100
4. 00010 of stroke
5. 00001 is a fold of
S4.2: matching the probability distribution of the stroke segment sequence obtained in the step S3 with a plurality of coded Chinese characters with the same stroke, and calculating the cross entropy of the two, wherein the cross entropy is used for measuring the difference between the two probability distributions and reflecting the similarity between the input Chinese character and the coded Chinese character:
wherein p isi(x) And q isi(x) Respectively representing the probability distribution of the encoded Chinese characters and the probability distribution of the input Chinese characters, wherein H (p, q) represents the similarity between characters based on cross entropy, and the cross entropy is a value calculated based on the input Chinese characters and all Chinese characters with m strokes in an encoded Chinese character library;
s4.3: the same-stroke Chinese character with the minimum cross entropy is the final predicted Chinese character.
In order to solve the technical problem, the invention adopts another technical scheme that: the recognition system for the Chinese eye writing signal recognition method based on the EOG comprises the following steps:
the eye movement signal preprocessing module is used for carrying out band-pass filtering on the original multi-lead eye movement data to obtain template data and Chinese character stroke data after noise is eliminated;
the stroke segment dividing module is used for dividing continuous template data and Chinese character stroke data into template stroke segment data and Chinese character stroke segment data by a sliding window detection blinking method;
the stroke classification module is used for carrying out stroke identification on the obtained template stroke segment data and Chinese character stroke segment data through a classifier based on DTW (dynamic time warping) and processing an identification result by using a softmax algorithm to obtain probability distribution of an identified stroke sequence;
and the Chinese character coding and matching module is used for establishing a Chinese character library, carrying out one-hot coding, matching the probability distribution of the stroke segment sequence obtained by stroke classification with the Chinese character with the same stroke in coding, and calculating the cross entropy of the two Chinese characters, wherein the Chinese character with the same stroke with the minimum cross entropy is the final predicted Chinese character.
The invention has the beneficial effects that:
(1) the invention realizes the recognition of the eye-writing strokes and Chinese characters by detecting and recognizing the unit eye movement signals based on the EOG, and divides the Chinese eye-writing data based on the electro-oculogram into five template stroke segment data of horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke segment data and Chinese character stroke segment data, so that the expansion of the recognition Chinese character set is simpler;
(2) the recognition system is little influenced by light change, light in calculation weight and low in cost, and the recognition method can provide rich control commands, so that the recognition system can be used as wearable eye-writing equipment for long-term application to participate in experiments and lives of subjects and can also be used as another communication mode for people with inconvenient hands and feet;
(3) the recognition method of the Chinese eye writing signal recognition system based on the EOG has the advantages of high recognition accuracy of the Chinese character, strong expansion capability of a corresponding Chinese character library, wide application prospect and the like.
Drawings
FIG. 1 is a flow chart of the Chinese eye writing signal recognition method based on EOG according to the present invention;
FIG. 2 is a distribution diagram of electrodes during acquisition;
FIG. 3 is a diagram of an eye movement pattern and corresponding original EOG waveform for five basic strokes according to the present invention;
FIG. 4 is a schematic diagram of a single experimental paradigm for collecting Chinese character data in accordance with the present invention;
FIG. 5 is a diagram of an example of a warping path of the present invention;
FIG. 6 is a block diagram showing the construction of an experimental apparatus according to the present invention;
FIG. 7 is a diagram of the stroke detection results of the randomly selected Chinese character "of the present invention;
FIG. 8 is a schematic diagram of a confusion matrix for all Chinese strokes in accordance with the present invention;
FIG. 9 is a chart of the results of Chinese character recognition in the random experiment of the present invention.
The parts in the drawings are numbered as follows: 1. bioelectrode, 2, signal amplifier, 3, EOD acquisition computer, 4, computer screen.
Detailed Description
The following detailed description of the preferred embodiments of the present invention, taken in conjunction with the accompanying drawings, will make the advantages and features of the invention easier to understand by those skilled in the art, and thus will clearly and clearly define the scope of the invention.
Referring to fig. 1, an embodiment of the present invention includes:
a Chinese eye writing signal recognition method based on EOG comprises the following steps:
s1: acquiring Chinese eye writing data based on an electro-oculogram and preprocessing the data;
the preprocessing process is to carry out band-pass filtering on the original multi-lead electro-ocular data, and the cut-off frequency of the band-pass filtering is 0.1-8 Hz.
S2: dividing the preprocessed data into template data and Chinese character stroke data: the template data is the template stroke segment data which is obtained by dividing continuous stroke template data into five templates, namely horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke, by adopting a sliding window blink detection method; the Chinese character stroke data is the preprocessed original multi-lead data and is divided into Chinese character stroke segment data by a sliding window detection blinking method;
in step S2, the sliding window blink detection method includes the steps of:
s2.1: taking the potential difference between the collecting electrodes HEOR and HEOL as a horizontal EOG signal (EOGh), taking the difference between the collecting electrodes VEOU and VEOD as a vertical EOG signal (EOGv), and processing the filtered vertical EOG signal through a sliding window;
s2.2: setting a dynamic threshold amp to be initialized to 225+ M, wherein M is the result of averaging all EOG amplitude values in a sliding window; comparing the amplitude of each EOG sample in the current sliding window with a dynamic threshold amp, if higher than amp, marking the corresponding point in the blink segment as a blink point, otherwise a non-blink point, and then updating the data in the sliding window by moving the sample points one by one;
s2.3: repeating step S2.2 until the sliding window moves to the end of the EOG signal;
s2.4: when the continuous EOG signal is detected, the blink segment is used for dividing the continuous EOG signal into a series of stroke segments, and therefore template stroke segment data and Chinese character stroke segment data are obtained.
The blink segment is an EOG signal data segment from one blink ending point to the next blink starting point.
S3: the method comprises the steps that five template stroke segments of horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke and Chinese character stroke segment data are all sent to a classifier based on DTW (Dynamic Time Warping) for stroke recognition, and recognition results are processed through a softmax algorithm to obtain probability distribution of recognized stroke segment sequences; the method comprises the following specific steps:
s3.1: suppose there is a Chinese character with a total stroke number of m, siRepresents the ith stroke, tjRepresents the jth template (i is more than or equal to 1 and less than or equal to m, j is more than or equal to 1 and less than or equal to 5) and each stroke(s) of the Chinese characteri) Match 5 template strokes and calculate similarity using DTW method:
p(si，tj)＝1-dtw(si，tj)/max1≤k≤5(dtw(si，tk)) (1)
in equation (1), dtw(s)i，tj) Is the cumulative distance, p(s), calculated by the DTW methodi，tj) Representing strokes siIs classified as a template tjProbability of reflecting Chinese character stroke segment siDrawing segment t with templatejThe greater its value, the greater the similarity between them, the stroke siAnd template tjThe more similar.
S3.2: p(s) of all strokes of Chinese characteri，tj) Are combined to obtain a similarity matrix P:
pi＝[p(si，t1)，p(si，t2)，…，p] (3)
wherein p isiIs the probability distribution of the ith stroke divided into five template strokes; for example, when pi＝[0.15，0.15，0.2，0.1，0.4]Then, the ith stroke is divided into 'B';
s3.3: because the similarity calculated by DTW is not probability distribution, the similarity is processed by adopting a softmax algorithm method, so that the similarity calculated by DTW becomes a probability distribution matrix Q, as shown in formulas (4) and (5):
in equation (5), Q represents the recognition probability distribution matrix of the m strokes of a Chinese character.
S4: and (4) establishing a Chinese character library and using one-hot coding to match the probability distribution of the stroke segment sequence obtained in the step (S3) with the coded Chinese character with the same stroke to obtain the final predicted Chinese character. The method comprises the following specific steps:
s4.1: selecting 100 common Chinese characters to establish a Chinese character library and using one-hot coding, wherein the legal combination of the values in the one-hot method only has one high order (1) and all other low orders (0). All Chinese characters are composed of five basic strokes of horizontal, vertical, left falling, right falling and turning, and the five types of strokes are expressed as one-hot:
1. cross section [ one ] 10000
2. 01000 in vertical axis
3. (iii) 00100
4. 00010 of stroke
5. 00001 is a fold of
S4.2: matching the probability distribution of the stroke segment sequence obtained in the step S3 with a plurality of coded Chinese characters with the same stroke, calculating the cross entropy of the two, wherein the quality of the language model is usually measured by the cross entropy and the complexity, the cross entropy is used for measuring the difference between the two probability distributions, and the similarity between the input Chinese character and the coded Chinese character is reflected:
wherein p isi(x) And q isi(x) Respectively representing the probability distribution of the encoded Chinese character and the probability distribution of the input Chinese character, and H (p, q) represents the similarity between the characters based on the cross entropy, which isThe smaller the value of (c), the more similar between the two characters. The cross entropy is a value calculated based on all Chinese characters with m strokes in the input Chinese character and the encoding Chinese character library;
s4.3: the same-stroke Chinese character with the minimum cross entropy is the final predicted Chinese character.
The recognition system of the Chinese eye writing signal recognition method based on the EOG comprises an eye movement signal preprocessing module, a stroke segment segmentation module, a stroke classification module and a Chinese character coding and matching module.
The eye movement signal preprocessing module is used for carrying out band-pass filtering on the original multi-lead eye movement data to obtain template data and Chinese character stroke data after noise is eliminated;
the stroke segment dividing module is used for dividing continuous template data and Chinese character stroke data into template stroke segment data and Chinese character stroke segment data by a sliding window detection blink method;
the stroke classification module is used for carrying out stroke identification on the obtained template stroke segment data and Chinese character stroke segment data through a classifier based on DTW (dynamic time warping) and processing an identification result by using a softmax algorithm to obtain probability distribution of an identified stroke sequence;
the Chinese character encoding and matching module is used for establishing a Chinese character library, carrying out one-hot encoding, matching probability distribution of stroke segment sequences obtained by stroke classification with the encoded Chinese characters with the same stroke, and calculating cross entropy of the two Chinese characters, wherein the Chinese character with the same stroke and the minimum cross entropy is the final predicted Chinese character.
Referring to fig. 1, which illustrates the overall framework of the algorithm in this embodiment, we first pre-process the raw EOG signal using a band-pass digital filter in order to suppress the noise interference. Then, by detecting a blink signal using a sliding window technique, valid eye movement segments are determined from the different stroke segments. On the basis, 5 basic stroke templates are established according to the characteristics of the Chinese characters. Considering the complexity of calculation, the DTW algorithm is adopted to classify EOG segments of different strokes. Finally, the stroke sequence is matched with the coded Chinese character to obtain a recognition result.
Referring to fig. 2, the electrode distribution during acquisition is shown. EOG data in the horizontal direction is acquired using HEOR and HEOL, while EOG data in the vertical direction is acquired by HEOU and HEOD. A1 and GND are reference and ground electrodes, respectively. In this example, a total of 6 bioelectrodes were used.
Referring to FIG. 3, the eye movement pattern of the five basic strokes and the corresponding design of the original EOG waveform in this embodiment are illustrated. 1-5: representing 5 basic strokes. (a) To (e): 5 eyeball movement patterns designed based on five basic strokes are shown. (h1) - (h5) and (v1) to (v 5): respectively, a preprocessed horizontal EOG signal and a vertical EOG signal.
Referring to fig. 4, the design of a single experimental paradigm for collecting chinese character data in this embodiment is illustrated: two types of eye-writing data, namely template data and simplified Chinese character eye-writing data, are collected. First, the subject should relax for 10 minutes to accommodate the course of the experiment and turn on the eye-writing system after pressing the start button. The subject is asked to focus on the center of the screen for about 2 seconds during the experiment, and then a prompt picture is displayed on the screen after the blinking motion is finished and the subject is asked to perform an eye writing activity, and the subject blinks again after the completion as the end of the stroke.
Referring to fig. 5, an example of a DTW warping path in this embodiment is illustrated, where the abscissa represents all the frames of the test model, the ordinate represents all the frames of the reference model, i.e., t (n) represents the template stroke data (total n frames), and r (m) represents the chinese character stroke data (total m frames). The points (N, M) represent the minimum distances D (T) (N), R (M), i.e. the minimum distance between each point of the sequence T and each point of R, i.e. the maximum similarity, calculated by the DTW algorithm, point (N, M) being the coordinates of the last point, N, M being the lengths of the two electro-ocular signals, respectively.
Referring to fig. 6, which illustrates the introduction of the experimental apparatus in this embodiment, the bioelectrode 1 disposed on the head of the subject was connected to the signal amplifier 2, and the other end of the signal amplifier 2 was connected to the EOG acquisition computer 3, and a total of 10 subjects (6 women and 4 men) were involved in the experiment and had normal hearing, right-handed writing, and normal eyesight. Prior to the experiment, the subject was asked to sit approximately 40 cm in front of the computer screen 4, the height of the center of the screen 4 being approximately the same as the subject's eyeball height. During this process, the subject should remain physically stable as much as possible. All cueing pictures were displayed randomly with the duration of relaxation fixed at 2 seconds and the duration of writing dependent on the subject. When the prompt disappears on the screen 4, the subject should blink again to mark the end point. Finally, a relaxation time of 2s was scheduled to better perform the next experiment. It should be noted that the subject needs to click the mouse to manually switch the cueing picture before the next trial.
Referring to fig. 7, the stroke detection result of the randomly selected chinese character "of" in the present embodiment is explained. (a) And (c): are filtered vertical and horizontal EOG data, (b) and (d): is a stroke segment after blink segmentation. The corresponding stroke sequence is shown in (e).
Referring to fig. 8, a confusion matrix of all chinese strokes in the present embodiment is illustrated, where the strokes on the left side of the matrix represent predicted strokes, the strokes on the top represent real strokes, correct stroke recognition results are displayed on the diagonal lines, and errors of the confusion strokes are displayed on the non-diagonal lines. On average, the maximum and minimum stroke confusion errors are 0.13% and 0%, respectively, and the recognition accuracy of all strokes is up to 93.4%.
Referring to fig. 9, the recognition result of the chinese character by the random experiment in the present embodiment is explained. The line below each Chinese character represents the sequence of eye strokes, and Y and N in brackets indicate that the Chinese character was recognized correctly or incorrectly. As can be seen from the figure, most of the Chinese characters were correctly recognized, and the average recognition rate of 10 subjects was 94%. The experimental result shows that the proposed stroke recognition algorithm has good performance.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes performed by the present specification and drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (5)
1. A Chinese eye writing signal recognition method based on EOG comprises the following steps:
s1: acquiring Chinese eye writing data based on an electro-oculogram and preprocessing the data;
s2: dividing the preprocessed data into template data and Chinese character stroke data:
the template data is the template stroke segment data which is obtained by dividing continuous stroke template data into five templates, namely horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke, by adopting a sliding window blink detection method;
the Chinese character stroke data is the preprocessed original multi-lead data and is divided into Chinese character stroke segment data by a sliding window detection blinking method;
s3: the method comprises the steps that five template stroke segments of horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke segment data and Chinese character stroke segment data are all sent to a DTW-based classifier for stroke recognition, and a recognition result is processed by a softmax algorithm to obtain probability distribution of a recognized stroke segment sequence; the method comprises the following specific steps:
s3.1: suppose there is a Chinese character with a total stroke number of m, siRepresents the ith stroke, tjRepresents the jth template (i is more than or equal to 1 and less than or equal to m, j is more than or equal to 1 and less than or equal to 5) and each stroke(s) of the Chinese characteri) Match 5 template strokes and calculate similarity using DTW method:
p(si，tj)＝1-dtw(si，tj)/max1≤k≤5(dtw(si，tk)) (1)
in equation (1), dtw(s)i，tj) Is the cumulative distance, p(s), calculated by the DTW methodi，tj) Representing strokes siIs classified as a template tjProbability of reflecting Chinese character stroke segment siDrawing segment t with templatejThe similarity between them;
s3.2: p(s) of all strokes of Chinese characteri，tj) Are combined to obtain a similarity matrix P:
pi＝[p(si，t1)，p(si，t2)，...，p] (3)
wherein p isiIs the probability distribution of the ith stroke divided into five template strokes;
s3.3: and (3) processing the similarity by adopting a softmax algorithm method, so that the similarity calculated by DTW becomes a probability distribution matrix Q, as shown in formulas (4) and (5):
in the formula (5), Q represents the recognition probability distribution matrix of m strokes of the Chinese character;
s4: and (4) establishing a Chinese character library and using one-hot coding to match the probability distribution of the stroke segment sequence obtained in the step (S3) with the coded Chinese character with the same stroke to obtain the final predicted Chinese character.
2. An EOG-based Chinese eye writing signal recognition method as claimed in claim 1, wherein the preprocessing in step S1 is to band-pass filter the original multi-lead eye electrical data, the band-pass filter cut-off frequency is 0.1-8 Hz.
3. An EOG based Chinese eye writing signal recognition method as claimed in claim 1 wherein the sliding window detection blinking method in step S2 comprises the steps of:
s2.1: taking the potential difference between the collecting electrodes HEOR and HEOL as a horizontal EOG signal, taking the difference between the collecting electrodes VEOU and VEOD as a vertical EOG signal, and processing the filtered vertical EOG signal through a sliding window;
s2.2: setting and initializing a dynamic threshold amp, wherein the value range of the dynamic threshold amp is (180+ M) - (250+ M), wherein M is the result of averaging all EOG amplitude values in a sliding window; comparing the amplitude of each EOG sample in the current sliding window with a dynamic threshold amp, if higher than amp, marking the corresponding point in the blink segment as a blink point, otherwise a non-blink point, and then updating the data in the sliding window by moving the sample points one by one;
s2.3: repeating step S2.2 until the sliding window moves to the end of the EOG signal;
s2.4: when the continuous EOG signal is detected, the blink segment is used for dividing the continuous EOG signal into a series of stroke segments, and therefore template stroke segment data and Chinese character stroke segment data are obtained.
4. An EOG based Chinese eye writing signal identifying method as claimed in claim 1, wherein the step S4 includes the specific steps of:
s4.1: establishing a Chinese character font library and coding by one-hot, wherein all Chinese characters are composed of five basic strokes of horizontal stroke, vertical stroke, left falling stroke, right falling stroke and turning stroke, and the five types of strokes are expressed by one-hot:
1. transverse [ one ]:10000
2. A vertical [ I ]:01000
3. Skimming [ sic ]:00100
4. Left-falling stroke [ left-falling stroke ]:00010
5. Folding [ B ]:00001
S4.2: matching the probability distribution of the stroke segment sequence obtained in the step S3 with a plurality of coded Chinese characters with the same stroke, and calculating the cross entropy of the two, wherein the cross entropy is used for measuring the difference between the two probability distributions and reflecting the similarity between the input Chinese character and the coded Chinese character:
wherein p isi(x) And q isi(x) Respectively representing the probability distribution of the encoded Chinese characters and the probability distribution of the input Chinese characters, wherein H (p, q) represents the similarity between characters based on cross entropy, and the cross entropy is a value calculated based on the input Chinese characters and all Chinese characters with m strokes in an encoded Chinese character library;
s4.3: the same-stroke Chinese character with the minimum cross entropy is the final predicted Chinese character.
5. An identification system of a chinese eye writing signal identification method based on EOG as claimed in claim 1, comprising:
the eye movement signal preprocessing module is used for carrying out band-pass filtering on the original multi-lead eye movement data to obtain template data and Chinese character stroke data after noise is eliminated;
the stroke segment dividing module is used for dividing continuous template data and Chinese character stroke data into template stroke segment data and Chinese character stroke segment data by a sliding window detection blinking method;
the stroke classification module is used for carrying out stroke identification on the obtained template stroke segment data and Chinese character stroke segment data through a DTW-based classifier and processing an identification result by using a soffmax algorithm to obtain probability distribution of an identified stroke sequence;
and the Chinese character coding and matching module is used for establishing a Chinese character library, carrying out one-hot coding, matching the probability distribution of the stroke segment sequence obtained by stroke classification with the Chinese character with the same stroke in coding, and calculating the cross entropy of the two Chinese characters, wherein the Chinese character with the same stroke with the minimum cross entropy is the final predicted Chinese character.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811025755.1A CN109308118B (en) | 2018-09-04 | 2018-09-04 | Chinese eye writing signal recognition system based on EOG and recognition method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811025755.1A CN109308118B (en) | 2018-09-04 | 2018-09-04 | Chinese eye writing signal recognition system based on EOG and recognition method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109308118A CN109308118A (en) | 2019-02-05 |
| CN109308118B true CN109308118B (en) | 2021-12-14 |
Family
ID=65224328
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811025755.1A Active CN109308118B (en) | 2018-09-04 | 2018-09-04 | Chinese eye writing signal recognition system based on EOG and recognition method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109308118B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111569248A (en) * | 2020-05-25 | 2020-08-25 | 福州大学 | Treatment method and treatment system for patients with autism |
| CN112578682A (en) * | 2021-01-05 | 2021-03-30 | 陕西科技大学 | Intelligent obstacle-assisting home system based on electro-oculogram control |
| CN113222867B (en) * | 2021-04-16 | 2022-05-20 | 山东师范大学 | Image data enhancement method and system based on multi-template image |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103559894A (en) * | 2013-11-08 | 2014-02-05 | 安徽科大讯飞信息科技股份有限公司 | Method and system for evaluating spoken language |
| EP2776454A1 (en) * | 2011-11-09 | 2014-09-17 | Merz Pharma GmbH & Co. KGaA | Modified neurotoxins with poly-glycine and uses thereof |
| CN107037883A (en) * | 2017-04-13 | 2017-08-11 | 安徽大学 | A kind of mixing brain machine interface system and method based on Mental imagery |
| CN107480716A (en) * | 2017-08-15 | 2017-12-15 | 安徽大学 | Method and system for identifying saccade signal by combining EOG and video |
| CN108269573A (en) * | 2017-01-03 | 2018-07-10 | 蓝盾信息安全技术有限公司 | Speaker Recognition System based on vector quantization and gauss hybrid models |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1347022A (en) * | 2001-05-06 | 2002-05-01 | 张荣民 | Chinese-character characteristic character input system |
| US20100069780A1 (en) * | 2008-03-19 | 2010-03-18 | Ocz Technology Group, Inc. | Method and apparatus for using biopotentials for simultaneous multiple control functions in computer systems |
| CN105988567B (en) * | 2015-02-12 | 2023-03-28 | 北京三星通信技术研究有限公司 | Handwritten information recognition method and device |
-
2018
- 2018-09-04 CN CN201811025755.1A patent/CN109308118B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2776454A1 (en) * | 2011-11-09 | 2014-09-17 | Merz Pharma GmbH & Co. KGaA | Modified neurotoxins with poly-glycine and uses thereof |
| CN103559894A (en) * | 2013-11-08 | 2014-02-05 | 安徽科大讯飞信息科技股份有限公司 | Method and system for evaluating spoken language |
| CN108269573A (en) * | 2017-01-03 | 2018-07-10 | 蓝盾信息安全技术有限公司 | Speaker Recognition System based on vector quantization and gauss hybrid models |
| CN107037883A (en) * | 2017-04-13 | 2017-08-11 | 安徽大学 | A kind of mixing brain machine interface system and method based on Mental imagery |
| CN107480716A (en) * | 2017-08-15 | 2017-12-15 | 安徽大学 | Method and system for identifying saccade signal by combining EOG and video |
Non-Patent Citations (2)
| Title |
|---|
| Electrooculography-based continuous eye-writing recognition system for efficient assistive communication systems;Fuming Fang等;《PLOS ONE》;20180209;第13卷;参见正文第1页-第12页及附图6-7 * |
| 基于笔划特征和MCLRNN模型的联机手写藏文识别;王维兰等;《计算机工程与应用》;20080511;参见正文第2页-第4页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109308118A (en) | 2019-02-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220011864A1 (en) | Systems, methods, apparatuses and devices for detecting facial expression and for tracking movement and location in at least one of a virtual and augmented reality system | |
| Shan et al. | A Simple Convolutional Neural Network for Accurate P300 Detection and Character Spelling in Brain Computer Interface. | |
| CN109308118B (en) | Chinese eye writing signal recognition system based on EOG and recognition method thereof | |
| Chen et al. | Eyebrow emotional expression recognition using surface EMG signals | |
| Kocejko et al. | Eye mouse for disabled | |
| CN107480716B (en) | Method and system for identifying saccade signal by combining EOG and video | |
| CN108904163A (en) | wheelchair control method and system | |
| KR101745774B1 (en) | Method and System for Recognition of Eye-Character based on Tracking Technique of Electro Oculogram Gaze | |
| CN116880691A (en) | Brain-computer interface interaction method based on handwriting track decoding | |
| CN107480635B (en) | Glance signal identification method and system based on bimodal classification model fusion | |
| Cheng et al. | Remote heart rate measurement from near-infrared videos based on joint blind source separation with delay-coordinate transformation | |
| Lv et al. | Design and implementation of an eye gesture perception system based on electrooculography | |
| Enikeev et al. | Recognition of sign language using leap motion controller data | |
| Ding et al. | A robust online saccadic eye movement recognition method combining electrooculography and video | |
| KR101795723B1 (en) | Recognition of basic emotion in facial expression using implicit synchronization of facial micro-movements | |
| CN115509355A (en) | MI-BCI interaction control system and method under integrated vision | |
| KR101736403B1 (en) | Recognition of basic emotion in facial expression using implicit synchronization of facial micro-movements | |
| Arai | Relations between psychological status and eye movements | |
| Attivissimo et al. | Performance evaluation of image processing algorithms for eye blinking detection | |
| KR102155134B1 (en) | Apparatus and method for human identification using gait pattern based on emg | |
| Strumiłło et al. | A vision-based head movement tracking system for human-computer interfacing | |
| KR20200144996A (en) | Motor performance coashing system based on muscle synergy library and calculation method for muscle synergy characteristic based on muscle synergy library | |
| Kim et al. | A practical biosignal-based human interface applicable to the assistive systems for people with motor impairment | |
| CN110414619B (en) | EMG signal identification method | |
| Salih et al. | Eye Blinking for Command Generation Based on Deep Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |