Invited reviewBrain–computer interfaces for communication and control
Introduction
Many different disorders can disrupt the neuromuscular channels through which the brain communicates with and controls its external environment. Amyotrophic lateral sclerosis (ALS), brainstem stroke, brain or spinal cord injury, cerebral palsy, muscular dystrophies, multiple sclerosis, and numerous other diseases impair the neural pathways that control muscles or impair the muscles themselves. They affect nearly two million people in the United States alone, and far more around the world (Ficke, 1991, NABMRR, 1992, Murray and Lopez, 1996, Carter, 1997). Those most severely affected may lose all voluntary muscle control, including eye movements and respiration, and may be completely locked in to their bodies, unable to communicate in any way. Modern life-support technology can allow most individuals, even those who are locked-in, to live long lives, so that the personal, social, and economic burdens of their disabilities are prolonged and severe.
In the absence of methods for repairing the damage done by these disorders, there are 3 options for restoring function. The first is to increase the capabilities of remaining pathways. Muscles that remain under voluntary control can substitute for paralyzed muscles. People largely paralyzed by massive brainstem lesions can often use eye movements to answer questions, give simple commands, or even operate a word processing program; and severely dysarthric patients can use hand movements to produce synthetic speech (e.g. Damper et al., 1987, LaCourse and Hladik, 1990, Chen et al., 1999, Kubota et al., 2000). The second option is to restore function by detouring around breaks in the neural pathways that control muscles. In patients with spinal cord injury, electromyographic (EMG) activity from muscles above the level of the lesion can control direct electrical stimulation of paralyzed muscles, and thereby restore useful movement (Hoffer et al., 1996, Kilgore et al., 1997, Ferguson et al., 1999).
The final option for restoring function to those with motor impairments is to provide the brain with a new, non-muscular communication and control channel, a direct brain–computer interface (BCI) for conveying messages and commands to the external world. A variety of methods for monitoring brain activity might serve as a BCI. These include, besides electroencephalography (EEG) and more invasive electrophysiological methods, magnetoencephalography (MEG), positron emission tomography (PET), functional magnetic resonance imaging (fMRI), and optical imaging. However, MEG, PET, fMRI, and optical imaging are still technically demanding and expensive. Furthermore, PET, fMRI, and optical imaging, which depend on blood flow, have long time constants and thus are less amenable to rapid communication. At present, only EEG and related methods, which have relatively short time constants, can function in most environments, and require relatively simple and inexpensive equipment, offer the possibility of a new non-muscular communication and control channel, a practical BCI.
In the 7 decades since Hans Berger's original paper (Berger, 1929), the EEG has been used mainly to evaluate neurological disorders in the clinic and to investigate brain function in the laboratory; and a few studies have explored its therapeutic possibilities (e.g. Travis et al., 1975, Kuhlman, 1978, Elbert et al., 1980, Rockstroh et al., 1989, Rice et al., 1993, Sterman, 2000). Over this time, people have also speculated that the EEG could have a fourth application, that it could be used to decipher thoughts, or intent, so that a person could communicate with others or control devices directly by means of brain activity, without using the normal channels of peripheral nerves and muscles. This idea has appeared often in popular fiction and fantasy (such as the movie ‘Firefox’ in which an airplane is controlled in part by the pilot's EEG (Thomas, 1977)). However, EEG-based communication attracted little serious scientific attention until recently, for at least 3 reasons.
First, while the EEG reflects brain activity, so that a person's intent could in theory be detected in it, the resolution and reliability of the information detectable in the spontaneous EEG is limited by the vast number of electrically active neuronal elements, the complex electrical and spatial geometry of the brain and head, and the disconcerting trial-to-trial variability of brain function. The possibility of recognizing a single message or command amidst this complexity, distortion, and variability appeared to be extremely remote. Second, EEG-based communication requires the capacity to analyze the EEG in real-time, and until recently the requisite technology either did not exist or was extremely expensive. Third, there was in the past little interest in the limited communication capacity that a first-generation EEG-based BCI was likely to offer.
Recent scientific, technological, and societal events have changed this situation. First, basic and clinical research has yielded detailed knowledge of the signals that comprise the EEG. For the major EEG rhythms and for a variety of evoked potentials, their sites and mechanisms of origin and their relationships with specific aspects of brain function, are no longer wholly obscure. Numerous studies have demonstrated correlations between EEG signals and actual or imagined movements and between EEG signals and mental tasks (e.g. Keirn and Aunon, 1990, Lang et al., 1996, Pfurtscheller et al., 1997, Anderson et al., 1998, Altenmüller and Gerloff, 1999, McFarland et al., 2000a). Thus, researchers are in a much better position to consider which EEG signals might be used for communication and control, and how they might best be used. Second, the extremely rapid and continuing development of inexpensive computer hardware and software supports sophisticated online analyses of multichannel EEG. This digital revolution has also led to appreciation of the fact that simple communication capacities (e.g. ‘Yes’ or ‘No’, ‘On’ or ‘Off’) can be configured to serve complex functions (e.g. word processing, prosthesis control). Third, greatly increased societal recognition of the needs and potential of people with severe neuromuscular disorders like spinal cord injury or cerebral palsy has generated clinical, scientific, and commercial interest in better augmentative communication and control technology. Development of such technology is both the impetus and the justification for current BCI research. BCI technology might serve people who cannot use conventional augmentative technologies; and these people could find even the limited capacities of first-generation BCI systems valuable.
In addition, advances in the development and use of electrophysiological recording methods employing epidural, subdural, or intracortical electrodes offer further options. Epidural and subdural electrodes can provide EEG with high topographical resolution, and intracortical electrodes can follow the activity of individual neurons (Schmidt, 1980, Ikeda and Shibbasaki, 1992, Heetderks and Schmidt, 1995, Levine et al., 1999, Levine et al., 2000, Wolpaw et al., 2000a). Furthermore, recent studies show that the firing rates of an appropriate selection of cortical neurons can give a detailed picture of concurrent voluntary movement (e.g. Georgopoulos et al., 1986, Schwartz, 1993, Chapin et al., 1999, Wessberg et al., 2000). Because these methods are invasive, the threshold for their clinical use would presumably be higher than for methods based on scalp-recorded EEG activity, and they would probably be used mainly by those with extremely severe disabilities. At the same time, they might support more rapid and precise communication and control than the scalp-recorded EEG.
This review summarizes the current state of BCI research with emphasis on its application to the needs of those with severe neuromuscular disabilities. In order to address all current BCI research, it includes approaches that use standard scalp-recorded EEG as well as those that use epidural, subdural, or intracortical recording. While all these present-day BCIs use electrophysiological methods, the basic principles of BCI design and operation discussed here should apply also to BCIs that use other methods to monitor brain activity (e.g. MEG, fMRI). The next sections describe the essential elements of any BCI and the several categories of electrophysiological BCIs, review current research, consider prospects for the future, and discuss the issues most important for further BCI development and application.
Section snippets
Dependent and independent BCIs
A BCI is a communication system in which messages or commands that an individual sends to the external world do not pass through the brain's normal output pathways of peripheral nerves and muscles. For example, in an EEG-based BCI the messages are encoded in EEG activity. A BCI provides its user with an alternative method for acting on the world. BCIs fall into two classes: dependent and independent.
A dependent BCI does not use the brain's normal output pathways to carry the message, but
Present-day BCIs
While many studies have described electrophysiological or other measures of brain function that correlate with concurrent neuromuscular outputs or with intent and might therefore function in a BCI system, relatively few peer-reviewed articles have described human use of systems that satisfy the BCI definition given in Section 2.1 and illustrated in Fig. 1, systems that give the user control over a device and concurrent feedback from the device. These studies are reviewed here. Studies from the
The future of BCI-based communication and control: key issues
Non-muscular communication and control is no longer merely speculation. The studies reviewed in the previous section show that direct communication from the brain to the external world is possible and can serve useful purposes. At the same time, the reality does not yet match the fantasy (e.g. Thomas, 1977): BCIs are not yet able to fly airplanes and are not likely to be doing so anytime soon. Present independent BCIs in their best moments reach 25 bits/min. For those who have no voluntary
Conclusions
A BCI allows a person to communicate with or control the external world without using the brain's normal output pathways of peripheral nerves and muscles. Messages and commands are expressed not by muscle contractions but rather by electrophysiological phenomena such as evoked or spontaneous EEG features (e.g. SCPs, P300, mu/beta rhythms) or cortical neuronal activity. BCI operation depends on the interaction of two adaptive controllers, the user, who must maintain close correlation between his
Acknowledgements
Work in the authors' laboratories has been supported by the National Center for Medical Rehabilitation Research, National Institute of Child Health and Human Development, National Institutes of Health (NIH) in the USA, by the Deutsche Forschungsgemeinschaft (DFG) in Germany, and by the Fonds zur Förderung der wissenschaftlichen Forschung in Austria.
References (198)
- et al.
A characterization of a single-trial adaptive filter and its implementation in the frequency domain
Electroenceph clin Neurophysiol
(1989) The operant conditioning of the electrical activity in the brain as a method for controlling neural and mental processes
- et al.
Removal of ocular artifact from the EEG: a review
Neurophysiol Clin
(2000) - et al.
Hand-held text-to-speech device for the non-vocal disabled
J Biomed Eng
(1987) - et al.
Effects of anxiety on attentional allocation and task performance: an information processing analysis
Behav Res Ther
(1997) - et al.
The contingent negative variation and the late positive wave of the average evoked potential
Electroenceph clin Neurophysiol
(1970) - et al.
Biofeedback of slow cortical potentials
Electroenceph clin Neurophysiol
(1980) - et al.
Talking off the top of your head: toward a mental prothesis utilizing event-related brain potentials
Electroenceph clin Neurophysiol
(1988) - et al.
P300-like potentials in the normal monkey using classical conditioning and the auditory ‘oddball’ paradigm
Electroenceph clin Neurophysiol
(1986) - et al.
A new method for off-line removal of ocular artifact
Electroenceph clin Neurophysiol
(1983)
Frontal interhemispheric asymmetry: self regulation and individual differences in humans
Neurosci Lett
Statistical evaluation of slow brain potentials on the basis of single trials
Electroenceph clin Neurophysiol
Self-initiation of EEG-based communication in paralyzed patients
Clin Neurophysiol
The cone electrode: a long-term electrode that records from neurites grown onto its recording surface
J Neurosci Methods
Self-regulation of interhemispheric asymmetry in humans
Neurosci Lett
EEG feedback training of epileptic patients: clinical and electroencephalographic analysis
Electroenceph clin Neurophysiol
Electric and magnetic fields of the brain accompanying internal simulation of movement
Cogn Brain Res
Neural mechanisms underlying brain waves: from neural membranes to networks
Electroenceph clin Neurophysiol
Spatial filter selection for EEG-based communication
Electroenceph clin Neurophysiol
ERPs in response to different display parameters and implications for brain–computer interface systems
Soc Neurosci Abstr
Psychophysiology and the EEG
Artifact processing in computerized analysis of sleep EEG – a review
Neuropsychobiology
Multivariate autoregressive models for classification of spontaneous electroencephalographic signals during mental tasks
IEEE Trans Biomed Eng
Adaptive logic networks
Handbook of neural computation
Linear classification of low-resolution EEG patterns produced by imagined hand movements
IEEE Trans Rehabil Eng
A practical EMG-based human–computer interface for users with motor disabilities
J Rehabil Res Dev
A virtual reality testbed for brain–computer interface research
IEEE Trans Rehabil Eng
Uber das electrenkephalogramm des menchen
Arch Psychiatr Nervenkr
Slow potentials of the cerebral cortex and behavior
Physiol Rev
Slow cortical potentials: their origin, meaning, and clinical use
A spelling device for the paralyzed
Nature
Rain Man's revelations
Nature
The thought translation device (TTD) for completely paralyzed patients
IEEE Trans Rehabil Eng
Single-trial processing of event-related potentials using outlier information
IEEE Trans Biomed Eng
Brain–computer interface research at the Neil Squire Foundation
IEEE Trans Rehabil Eng
Avoidance training of hippocampal theta waves in flaxedilized dogs and its relation to skeletal movement
J Comp Physiol Psychol
The direct control of neural processes by reward and punishment
Am Sci
Rehabilitation management in neuromuscular disease
J Neurol Rehabil
Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex
Nat Neurosci
The mu rhythm
The new design of an infrared-controlled human–computer interface for the disabled
IEEE Trans Rehabil Eng
Event-related potentials: an introduction
Cortical control of motor prostheses: using the cord-transected baboon as the primate model for human paraplegia
Adv Neurol
Discriminative conditioning of hippocampal electrical activity in curarized dogs
Comm Behav Bio
Issues in cochlear prosthetics from an international survey of opinions
Int J Technol Assess Health Care
Occipital alpha rhythm, eye position and lens accommodation
Nature
Motor areas of the cerebral cortex
J Clin Neurophysiol
Presidential address, 1980. Surprise!…Surprise?
Psychophysiology
The mental prosthesis: assessing the speed of a P300-based brain–computer interface
IEEE Trans Rehabil Eng
Pyramidal tract activity associated with a conditioned hand movement in the monkey
J Neurophysiol
Cited by (6430)
Facilitating applications of SSVEP-BCI by effective Cross-Subject knowledge transfer
2024, Expert Systems with ApplicationsEEG-based motor imagery classification with quantum algorithms[Formula presented]
2024, Expert Systems with ApplicationsAn outlier detection based two-stage EEG artifact removal method using empirical wavelet transform and canonical correlation analysis
2024, Biomedical Signal Processing and ControlLGNet: Learning local–global EEG representations for cognitive workload classification in simulated flights
2024, Biomedical Signal Processing and ControlConvolutional neural networks reveal properties of reach-to-grasp encoding in posterior parietal cortex
2024, Computers in Biology and Medicine