GB2396421A

GB2396421A - Head-worn device measuring brain and facial muscle activity

Info

Publication number: GB2396421A
Application number: GB0229262A
Authority: GB
Inventors: Paul Laurence Reynolds; Stuart Murray Shutt
Original assignee: Orange SA; ORANGE PERSONAL COMM SERV Ltd; Orange Personal Communications Services Ltd
Current assignee: Orange SA
Priority date: 2002-12-16
Filing date: 2002-12-16
Publication date: 2004-06-23
Also published as: GB0229262D0

Abstract

Brain and facial muscle activity signals (EEG and EMG) are detected by a device worn on the head of a user 110, 210 and user perceptible signals are modified, generated or output. The head-worn device includes skin contact electrodes 122, 124, 126 mounted on spectacles 110 (at the ear 120, temple 118 and nose-piece 116) or 222, 224, 226 mounted on headphones 210 (at ear pieces 212 and headband 214). The measured signals may be transmitted to a terminal (e.g mobile phone, PDA, PC) via a wireless radio link where they are analysed and used to control the terminal. Also disclosed is the use of detected facial muscle activity to provide improvement in speech recognition.

Description

Terminal Control This invention relates to systems and methods of

controlling a terminal. More particularly, the invention relates to a system and method of controlling a terminal using brain-induced waves.

5 Systems and methods of detecting brain waves and controlling devices using brain-induced waves are known in the art.

Brain waves or electrical activity of the brain are usually detected with an array of electrodes attached to the scalp. Electroencephalogram (EEG) is a technique for measuring the summated electrical potentials of the cerebral 10 neurons to obtain an electroencephalogram.

Clinical EEG examination systems usually use 21 electrodes. A careful placement of the electrodes is required to ensure a satisfactory measurement of the brain-induced signal. Often the so-called "International 10-20 System" arrangement is used which places the electrodes according to 15 percentages of the total distances between the bridge of the nose and the inion in the longitudinal direction and between the two ears in the lateral direction.

The scalp is prepared in order to achieve a low impedance between the electrodes and the skin.

US patent S,038,782 relates to a brain wave detection system for 20 generating an electroencephalogram. The system includes a plurality of electrodes integrated in a flexible hat, which is adapted to be placed on the head. In this way the electrodes do not need to be individually and manually positioned on the scalp.

In this patent the use of "dry" or "semidry" electrodes has been described. Special amplifiers are used to dynamically match the amplifier input impedance to the electrode impedance. In this way it is ensured that the recording of the brain activity is satisfactory independent of the actual 5 electrode impedance. By using these "dry" electrodes no skin preparation is required and either no or only minimal electrode gel needs to be applied.

PCT patent application WO 02/50625 relates to a system for controlling a computer using electrophysiological data, such as BEG signals, electromyographic (EMG) signals, electrooculographic (EOG) signals, as well 10 as other measurements. The electrophysiological data are detected using a plurality of sensors, and control signals are derived from the detected data.

EOG signals are derived from the standing electrical potential that exists between the cornea and the retina of the eye. If the eye moves, the standing electrical potential of the eye causes a change in the potential over 15 scalp areas around the eyes. This change in potential is relatively easily measurable. The EOG can provide discrete, identifiable and controllable signals based on blinks and eye movements in any of the directions up, down, left and right. The electrodes are typically positioned close to the eye, for example on 20 the temple above or below the eye.

EMG is the electrical activity generated by muscle fibres during contraction. Voluntary contraction of a muscle causes an increase in EMG activity which is sustained for the duration of the muscle contraction.

Muscular electrical signals are measured, which provide an identifiable signal which is controllable from each discrete muscle sampled.

Head and neck muscles are normally quite finely controllable, so facial muscles such as orbicularis oris (the circular muscle around the mouth) and 5 orbicularis oculi (that around each eye) are suitable and others could be identified experimentally.

Several usable "command" signals can be derived from each muscle by quantifying the strength of a muscle contraction by the "fullness" of its interference pattern. During contraction of a muscle, the muscle fibres 10 contract and as the strength of contraction is increased more fibres are recruited to contract. This is reflected in the EMG interference pattern which is patchy and intermittent during weak contraction but which becomes more continuous and "full" as more fibres are recruited until, at maximum volitional effort, all the fibres in the muscle are recruited and the interference 15 pattern is "full" with a large amplitude.

A commercial system for monitoring electrophysiological activity in order to control a computer is the Cyberlink_ device of Brain Actuated Technologies, Inc. The device is described to use electrophysiological signals derived from EEG, from FOG and from EMG signals.

20 The Cyberlink_ system provides a headband including three electrodes in order to measure a composite of brain wave (EEG, EOG and EMG) activity. The electrodes are placed on the headband in a way that they

are positioned on the user's forehead when the user wears the headband. Also in this system "dry" electrodes are used.

The electrodes in the headband are connected to an interface device. In the interface device, the signals detected with the electrodes are amplified, 5 filtered and digitised. The signals so processed are subsequently sent to a Personal Computer (PC) via a serial port. The signals may be used for hands free control of the mouse for the PC including the operation of left and right mouse click buttons, for controlling the PC's keyboard, for interactive video game control and also for music synthesis.

10 Similar systems using electrodes positioned on a headband include the Interactive Brainwave Visual Analyser (IBVA) produced by IBVA Technologies, Inc. and the Hands Free Controller produced by BioControl Systems, Inc. It is an object of the present invention to provide an improved device 15 for detecting brain-induced signals.

It is another object of the present invention to provide an improved system and method to control a terminal.

According to a first aspect of the present invention, there is provided a device which is usually worn on or around a user's head and used for a 20 modifying, generating or outputting visual or audible signals, and said device is being adapted to detect the user's brain activity and/or facial muscle activity.

In this way the device for detection of brain-induced signals is integrated in a device which is usually already worn by the user to perform another function, such as a pair of eyeglasses or headphones. By integrating the detection means for detecting brain activity into a device usually worn by 5 the user for performing another function, there is no need for a separate device in order to monitor the brain waves. Thus detection of brain activity can be performed less intrusively than by using a separate device which is dedicated for monitoring brain activity. User acceptability for such a device for detecting and monitoring brain-induced data is higher if no separate device 10 is required.

Preferably, the device is adapted to have in use at least two points of contact with the skin of the user's head.

In this way it is ensured that the device is still capable of detecting the brain-induced signals, which are, for example, required for controlling a 15 mobile terminal. Such a control is relatively secure, as the terminal is trained to respond to the brain activity signals determined from a particular user.

Preferably, the device is a pair of eyeglasses. In this document the term eyeglasses is used to include spectacles and sunglasses. In this way a user usually wearing spectacles or sunglasses may use the pair of eyeglasses 20 at the same time to detect brain activity, which may then be used to control a terminal device. Preferably, but not essential to the invention, spectacles include lenses to modify or correct visual signals detected by the person

wearing the pair of eyeglasses. In a similar manner, sunglasses include filters to modify the visual signals.

Preferably, integrating the detection means into a pair of eyeglasses ensures at least four points of contact with the skin of the user's head when the 5 glasses are worn, one on either side of the bridge and one behind either ear.

Preferably, the user's detected brain and facial muscle activity is used for controlling a terminal device.

Alternatively or in addition, the user's detected brain and facial muscle activity may also be used for speech recognition.

10 It is known that the accuracy of speech recognition systems can be enhanced by the use of a camera which monitors the movement of the user's mouth. However, the accuracy of speech recognition systems can alternatively or in addition be improved by using brain-induced data relating to the control of the muscles which are used for speaking. Some of the muscles which 15 control the movement of the mouth are anchored around the ear. By measuring the electrical signals which are stimulating these muscles and using this information in speech recognition systems, the system for recognising the user's speech can be improved.

According to another aspect of the present invention, there is provided 20 a mobile terminal for use in a mobile communications network, said terminal being adapted to be controlled by the user's brain-induced and facial muscle signals.

According to another aspect of the present invention, there is provided a terminal adapted to receive brain activity and/or facial muscle activity signals relating to the control of muscles involved in speaking, said terminal being further adapted to use said signals for speech recognition.

5 Further aspects and embodiments of the present invention will now be described by way of example only, with reference to the accompanying drawings, in which: Figure 1A schematically illustrates a mobile terminal in which the present invention can be implemented; 10 Figure 1B is a block diagram illustrating component parts of the mobile terminal shown in Figure 1A; Figure 2A is a block diagram illustrating component parts of a brain induced control device according to the prior art;

Figure 2B schematically illustrates the placement of an array of 15 electrodes on the user's skull for the brain-induced control device of Figure 2A; Figure 2C illustrates a headband for monitoring brain activity data according to the prior art;

Figure 3 is a schematic illustration of a pair of eyeglasses according to 20 one embodiment of the present invention.

Figure 4 is a block diagram illustrating component parts of the brain induced control device according to one embodiment of the present invention; and

Figure 5 is a schematic illustration of headphones according to another embodiment of the present invention.

Figure lA is a schematic illustration of a mobile communications terminal in which the present invention can be implemented, and Figure I B is 5 a block diagram illustrating component parts of the mobile terminal shown in Figure 1A.

Referring now to Figure 1B, the terminal 10 comprises a processor 22, radio communication means 24 for communicating with other devices via a mobile communications network, an antenna means 26, a memory 28 and a 10 user interface 30. Referring now to Figure 1A, the user interface of the mobile terminal is briefly described. The front surface 12 of the terminal 10 includes a display 14 for displaying information to the user and a keypad 16 for operating the terminal. User interface 30 further includes a microphone 18 and a speaker 19.

15 In the conventional GSM (Global System for Mobile Communications) system, each mobile terminal, such as a mobile telephone handset, is provided with a SIM (Subscriber Identity Module), which is inserted into the mobile station in order to allow the mobile terminal to receive services in a GSM network from a particular service provider. The 20 SIM stores information related to a particular subscription with a service provider. Referring now to Figure 2A, a schematic outline of a control device utilising brain activity according to the prior art is shown. The devices

typically include a sensor device 52 including a plurality of electrical electrodes to measure the electrical activity of the brain. The sensor device 52 is connected to an amplifier 54 for amplifying the brain activity signals. A processing device 56 such as a digital signal processor (DSP) is provided for 5 processing the signals such that the signals can be used to control a device.

Processing the signals includes converting the analog signals into digital signals by an analog-to-digital converter and filtering the signals.

Software running on the PC is used to analyse the signals and to identify a control signal corresponding to the identified signal or signal 10 pattern. In order to analyse the complex signals, tools like neural networks based on pattern recognition or the like may be used.

In order to monitor or measure the electrical signals, a large array of electrodes is usually used. Such an array includes recording electrodes 62 which are arranged on predetermined positions on the person's head 63. The 15 electrodes are connected to a processing unit 64, which amplifies and processes the signals as described above. The use of such an array is illustrated in Figure 2B.

Commercially available products for brain-induced control, such as for example the Cyberlink_ system, use headbands including a plurality of 20 electrodes for monitoring brain waves. Figure 2C illustrates a headband 70 including three electrodes 72 for detecting the brain waves. The electrodes are connected to a processing unit 74 for processing the detected signals. The headband 70 is worn around the user's head such that the electrodes 72 are

placed on the user's forehead. In this way the electrodes are in direct contact with the skin of the head to monitor the brain activity.

The disadvantage of the systems described is that a user needs to wear a special, usually cumbersome device including the electrodes in order to 5 detect the brain activity. Thus controlling a terminal device using brain activity is an intrusive method, and the acceptance to the user is therefore low.

Also, controlling a terminal using brain activity in this way needs pre arrangement, i.e. the user has to put on the electrodes or electrode device specifically in order to use brain-induced control. The method is thus not 10 readily available for a user without preparation.

The present invention aims to alleviate these disadvantages.

According to one embodiment of the present invention, a pair of eyeglasses is adapted to be used for detecting and monitoring brain activity.

Eyeglasses are already worn by many individuals. Thus a separate device like 15 a headband becomes superfluous for those people already wearing glasses.

Electrodes for detecting and monitoring brain activity are placed on the spectacles such that they are in direct contact with the user's skin in a position which is suitable for detecting brain-induced and facial muscle data.

The detected signals are then pre-processed such that the signals are suitable 20 for transmission to a terminal. The signals are transmitted using a wireless link. Referring now to Figure 3, a pair of eyeglasses is shown which is adapted for monitoring brain activity. The frame 110 includes two lens pieces

112, two temple pieces 118 and a bridge piece 114 between the two lens pieces, including two nose pads 116 for supporting the eyeglasses on either side of the nose when the eyeglasses are worn. Each temple piece includes an arcuate earpiece 120 on the end opposite to the lens piece 112.

5 Electrodes are positioned on the eyeglasses such that they are in direct contact with the user's skin. Electrodes 122 and 124 are placed on each nose pad 116 and each earpiece 120, respectively. Additional electrodes 126 may be positioned on the temple pieces 118, such that they are in contact with the user's temples.

10 The distance between both temple pieces 118 needs to be adjusted so that the electrodes 126 and 124 on the temple pieces 118 and ear pieces 120, respectively, come in contact with the user's temples and the skin behind the user's ears when the eyeglasses are in use.

The nose-pads 116 of bridge 114 are pressed against the user's nose by 15 the weight of eyeglasses 110 such that it is ensured that electrodes 122 are in contact with the user's skin when the eyeglasses 110 are in use.

Eyeglasses 110 further include a processor element 132 and a battery 134, each mounted to one of the two earpieces 120. All described elements relating to monitoring brain activity, i.e. the electrodes 122, 124 and 126 and 20 the battery 134 are connected to processor element 132 by wires laid internally within the frame 110.

Referring now to Figure 4, the individual elements of the brain activity monitoring device and the terminal are outlined.

The brain activity signals are monitored by electrodes 122, 124 and 126 and the signals are transmitted to processor element 132. Processor element 132 includes a microprocessor 136 and a chip 138 adapted to transmit signals using radio system, such as the Bluetooth_ protocol, to the terminal 5 device The terminal, which is to be controlled by the brain-induced signals, also includes a chip 142, which is adapted to receive signals using the Bluetooth_ protocol, an amplifier 144 and a processor 146. The terminal may for example be a mobile terminal for use in a cellular communications 10 network such as the terminal illustrated in Figure 1.

As the elements for monitoring brain-induced signals are included in eyeglasses 110, the signals need to be transmitted to the terminal 140 which is to be controlled by the signals. This is achieved by equipping both the eyeglasses 110 and also the terminal 140 with chips 138, 142 capable of 15 communicating via radio syatem such as the Bluetooth_ protocol.

The microprocessor pre-processes the signals received from electrodes 122, 124 and 126 in order to be used as brain-induced control signals. The signals are derived from the electrode readings by recording the difference between two electrodes. One electrode, typically one or both of electrodes 20 124 situated in use behind the user's ear are used as reference electrodes and the voltage difference between the reference electrode and the remaining electrodes is detected.

It is apparent that a common reference electrode may be used for monitoring a plurality of electrodes. In this way N channels can be monitored from N+ 1 electrodes.

The signals are pre-processed, for example by using pulse position 5 modulation, such that they are suitable for transmission via the radio system such as Bluetooth_ radio link. The so prepared signals may then be transmitted from the eyeglasses to mobile terminal via the radio system link.

The terminal receives the brain-induced signals with BluetoothTM element 142 and amplifies the signals in amplifier 144. Subsequently, the 10 signals are processed and analysed in processor 146 in order to identify command signals which are then used to control the terminal 140.

According to a second embodiment of the present invention, the elements relating to monitoring brain activity may be implemented in headphones instead of eyeglasses.

15 Figure 5 shows a schematic outline of headphones 210 comprising two earpieces 212 and a headpiece 214. The headphones 210 further include a plurality of electrodes 222, 224 and 226. One electrode 224 is positioned on the inner lower part of each of the earpieces 212, whereas two electrodes 222, are placed on the inner upper part of each of the earpieces 212. Further 20 electrodes 226 may be positioned on the inner side of headpiece 214.

If the headphones 210 are worn by a user, electrodes 224 are positioned behind the user's ear and may serve as reference electrodes.

Electrodes 222 are positioned around the upper part of the user's ear if the

headphones are worn, whereas electrodes 226 come in contact with the user's skull. The headphones 210 further include a processor element 232 including a Bluetooth_ radio link element and a battery 234.

5 Whilst in the above described embodiments a device for detecting brain activity is described to be provided with a chip for transmitting, using the Bluetooth_ protocol, the brain signals to the terminal which uses the signals, it is appreciated that alternatively other means of transmitting the signals may be used instead. These alternative means may include other 10 wireless radio links, such as links using the Infrared Data Association (IrDA) protocol. Whilst in the above described embodiments a terminal is described to be controlled by brain-induced control signals, it is appreciated that the brain activity data may alternatively be used for other purposes, as for example for 15 speech recognition.

It is known that the accuracy of speech recognition systems can be enhanced by the use of a camera which monitors the movement of the user's mouth. However, the accuracy of speech recognition systems can alternatively or in addition be improved by using brain-induced data relating to the control 20 of the muscles which are used for speaking.

As described above, the facial muscles such as the orbicularis oris are very precisely controllable. Some of the muscles which control the movement of the mouth are anchored around the ear. By placing the

electrodes close to the user's ear as described above, the electrical signals which are stimulating these muscles can be detected. By monitoring the EMG signals behind the user's ear or at other suitable positions, information can be derived on which muscles are used to form particular sounds and on how the 5 muscles are contracted. These signals may be used in addition to conventional speech recognition techniques to improve speech recognition systems. Whilst in the above described embodiments a mobile communications terminal is described to be controlled by brain-induced control signals, it is 10 appreciated that alternatively other devices, like a computer terminal or a personal digital assistant (PDA) device may be controlled.

Also, whilst in the above described embodiments eyeglasses and headphones has been used for implementing electrodes to detect brain waves, it is appreciated that alternatively other devices for modifying, generating or 15 outputting visual or audible signals may be used. Such device may be, for example, a binocular display worn similarly to the way conventional eyeglasses are worn.

It is to be understood that the embodiments described above are preferred embodiments only. Various features may be omitted, modified or 20 substituted by equivalents, without departing from the scope of the present invention.

Claims

CLAIMS:

1. A device for modifying, generating or outputting visual or audible signals for perception by a user, said device being adapted to be worn on or around the user's head, and said device further being adapted to detect 5 the user's brain activity and/or facial muscle activity.

2. A device according to claim l, adapted in use to have at least two points of contact to the skin of the user's head to detect the user's brain activity and/or facial muscle activity.

3. A device according to claim I or 2, said points of contact comprising a point close to the root of the nose and/or close to the user's ear to detect the user's brain and facial muscle activity.

15 4. A device according to claim 2 or 3, including electrodes at said points of contact for detecting brain activity and/or facial muscle activity.

5. A device according to any preceding claim, adapted to transmit brain and/or facial muscle activity and/or facial muscle activity signals to a 20 terminal device via a short range radio link.

6. A device according to any preceding claim, adapted to perform processing of brain activity and/or facial muscle activity signals.

7. A device according to any preceding claim, wherein said device is a pair of eyeglasses.

5 8. A device according to claim 7, wherein an electrode for detecting brain activity is positioned on an earpiece.

9. A device according to claim 7 or 8, wherein an electrode for detecting brain activity is positioned on the nose-piece of the bridge.

10. A device according to claim 7, 8 or 9, wherein an electrode for detecting brain activity and/or facial muscle activity is positioned on a temple piece. 15 11. A device according to any of claims I to 6, wherein said device is a headphone.

12. A device according to any of claims 1 to 6, wherein said device is a display worn on or around the user's head.

13. A mobile terminal for use in a mobile communications network, adapted to be controlled in dependence on the user's detected brain and/or facial muscle activity.

14. A mobile terminal according to claim 13, further comprising a processor for receiving brain-induced signals, analysing said signals and identifying control signals from said brain-induced signals for controlling a 5 function of said mobile terminal.

15. A mobile terminal according to claim 13 or 14, further adapted to communicate via a short range radio link for receiving brain-induced signals. 16. A terminal adapted to receive brain activity and/or facial muscle activity signals relating to the control of muscles involved in speaking, said terminal being further adapted to use said signals for speech recognition.

15 17. A device substantially as hereinbefore described with reference to the accompanying drawings.