Invited review
Brain–computer interfaces for communication and control

https://doi.org/10.1016/S1388-2457(02)00057-3Get rights and content

Abstract

For many years people have speculated that electroencephalographic activity or other electrophysiological measures of brain function might provide a new non-muscular channel for sending messages and commands to the external world – a brain–computer interface (BCI). Over the past 15 years, productive BCI research programs have arisen. Encouraged by new understanding of brain function, by the advent of powerful low-cost computer equipment, and by growing recognition of the needs and potentials of people with disabilities, these programs concentrate on developing new augmentative communication and control technology for those with severe neuromuscular disorders, such as amyotrophic lateral sclerosis, brainstem stroke, and spinal cord injury. The immediate goal is to provide these users, who may be completely paralyzed, or ‘locked in’, with basic communication capabilities so that they can express their wishes to caregivers or even operate word processing programs or neuroprostheses. Present-day BCIs determine the intent of the user from a variety of different electrophysiological signals. These signals include slow cortical potentials, P300 potentials, and mu or beta rhythms recorded from the scalp, and cortical neuronal activity recorded by implanted electrodes. They are translated in real-time into commands that operate a computer display or other device. Successful operation requires that the user encode commands in these signals and that the BCI derive the commands from the signals. Thus, the user and the BCI system need to adapt to each other both initially and continually so as to ensure stable performance. Current BCIs have maximum information transfer rates up to 10–25 bits/min. This limited capacity can be valuable for people whose severe disabilities prevent them from using conventional augmentative communication methods. At the same time, many possible applications of BCI technology, such as neuroprosthesis control, may require higher information transfer rates. Future progress will depend on: recognition that BCI research and development is an interdisciplinary problem, involving neurobiology, psychology, engineering, mathematics, and computer science; identification of those signals, whether evoked potentials, spontaneous rhythms, or neuronal firing rates, that users are best able to control independent of activity in conventional motor output pathways; development of training methods for helping users to gain and maintain that control; delineation of the best algorithms for translating these signals into device commands; attention to the identification and elimination of artifacts such as electromyographic and electro-oculographic activity; adoption of precise and objective procedures for evaluating BCI performance; recognition of the need for long-term as well as short-term assessment of BCI performance; identification of appropriate BCI applications and appropriate matching of applications and users; and attention to factors that affect user acceptance of augmentative technology, including ease of use, cosmesis, and provision of those communication and control capacities that are most important to the user. Development of BCI technology will also benefit from greater emphasis on peer-reviewed research publications and avoidance of the hyperbolic and often misleading media attention that tends to generate unrealistic expectations in the public and skepticism in other researchers. With adequate recognition and effective engagement of all these issues, BCI systems could eventually provide an important new communication and control option for those with motor disabilities and might also give those without disabilities a supplementary control channel or a control channel useful in special circumstances.

Introduction

Many different disorders can disrupt the neuromuscular channels through which the brain communicates with and controls its external environment. Amyotrophic lateral sclerosis (ALS), brainstem stroke, brain or spinal cord injury, cerebral palsy, muscular dystrophies, multiple sclerosis, and numerous other diseases impair the neural pathways that control muscles or impair the muscles themselves. They affect nearly two million people in the United States alone, and far more around the world (Ficke, 1991, NABMRR, 1992, Murray and Lopez, 1996, Carter, 1997). Those most severely affected may lose all voluntary muscle control, including eye movements and respiration, and may be completely locked in to their bodies, unable to communicate in any way. Modern life-support technology can allow most individuals, even those who are locked-in, to live long lives, so that the personal, social, and economic burdens of their disabilities are prolonged and severe.

In the absence of methods for repairing the damage done by these disorders, there are 3 options for restoring function. The first is to increase the capabilities of remaining pathways. Muscles that remain under voluntary control can substitute for paralyzed muscles. People largely paralyzed by massive brainstem lesions can often use eye movements to answer questions, give simple commands, or even operate a word processing program; and severely dysarthric patients can use hand movements to produce synthetic speech (e.g. Damper et al., 1987, LaCourse and Hladik, 1990, Chen et al., 1999, Kubota et al., 2000). The second option is to restore function by detouring around breaks in the neural pathways that control muscles. In patients with spinal cord injury, electromyographic (EMG) activity from muscles above the level of the lesion can control direct electrical stimulation of paralyzed muscles, and thereby restore useful movement (Hoffer et al., 1996, Kilgore et al., 1997, Ferguson et al., 1999).

The final option for restoring function to those with motor impairments is to provide the brain with a new, non-muscular communication and control channel, a direct brain–computer interface (BCI) for conveying messages and commands to the external world. A variety of methods for monitoring brain activity might serve as a BCI. These include, besides electroencephalography (EEG) and more invasive electrophysiological methods, magnetoencephalography (MEG), positron emission tomography (PET), functional magnetic resonance imaging (fMRI), and optical imaging. However, MEG, PET, fMRI, and optical imaging are still technically demanding and expensive. Furthermore, PET, fMRI, and optical imaging, which depend on blood flow, have long time constants and thus are less amenable to rapid communication. At present, only EEG and related methods, which have relatively short time constants, can function in most environments, and require relatively simple and inexpensive equipment, offer the possibility of a new non-muscular communication and control channel, a practical BCI.

In the 7 decades since Hans Berger's original paper (Berger, 1929), the EEG has been used mainly to evaluate neurological disorders in the clinic and to investigate brain function in the laboratory; and a few studies have explored its therapeutic possibilities (e.g. Travis et al., 1975, Kuhlman, 1978, Elbert et al., 1980, Rockstroh et al., 1989, Rice et al., 1993, Sterman, 2000). Over this time, people have also speculated that the EEG could have a fourth application, that it could be used to decipher thoughts, or intent, so that a person could communicate with others or control devices directly by means of brain activity, without using the normal channels of peripheral nerves and muscles. This idea has appeared often in popular fiction and fantasy (such as the movie ‘Firefox’ in which an airplane is controlled in part by the pilot's EEG (Thomas, 1977)). However, EEG-based communication attracted little serious scientific attention until recently, for at least 3 reasons.

First, while the EEG reflects brain activity, so that a person's intent could in theory be detected in it, the resolution and reliability of the information detectable in the spontaneous EEG is limited by the vast number of electrically active neuronal elements, the complex electrical and spatial geometry of the brain and head, and the disconcerting trial-to-trial variability of brain function. The possibility of recognizing a single message or command amidst this complexity, distortion, and variability appeared to be extremely remote. Second, EEG-based communication requires the capacity to analyze the EEG in real-time, and until recently the requisite technology either did not exist or was extremely expensive. Third, there was in the past little interest in the limited communication capacity that a first-generation EEG-based BCI was likely to offer.

Recent scientific, technological, and societal events have changed this situation. First, basic and clinical research has yielded detailed knowledge of the signals that comprise the EEG. For the major EEG rhythms and for a variety of evoked potentials, their sites and mechanisms of origin and their relationships with specific aspects of brain function, are no longer wholly obscure. Numerous studies have demonstrated correlations between EEG signals and actual or imagined movements and between EEG signals and mental tasks (e.g. Keirn and Aunon, 1990, Lang et al., 1996, Pfurtscheller et al., 1997, Anderson et al., 1998, Altenmüller and Gerloff, 1999, McFarland et al., 2000a). Thus, researchers are in a much better position to consider which EEG signals might be used for communication and control, and how they might best be used. Second, the extremely rapid and continuing development of inexpensive computer hardware and software supports sophisticated online analyses of multichannel EEG. This digital revolution has also led to appreciation of the fact that simple communication capacities (e.g. ‘Yes’ or ‘No’, ‘On’ or ‘Off’) can be configured to serve complex functions (e.g. word processing, prosthesis control). Third, greatly increased societal recognition of the needs and potential of people with severe neuromuscular disorders like spinal cord injury or cerebral palsy has generated clinical, scientific, and commercial interest in better augmentative communication and control technology. Development of such technology is both the impetus and the justification for current BCI research. BCI technology might serve people who cannot use conventional augmentative technologies; and these people could find even the limited capacities of first-generation BCI systems valuable.

In addition, advances in the development and use of electrophysiological recording methods employing epidural, subdural, or intracortical electrodes offer further options. Epidural and subdural electrodes can provide EEG with high topographical resolution, and intracortical electrodes can follow the activity of individual neurons (Schmidt, 1980, Ikeda and Shibbasaki, 1992, Heetderks and Schmidt, 1995, Levine et al., 1999, Levine et al., 2000, Wolpaw et al., 2000a). Furthermore, recent studies show that the firing rates of an appropriate selection of cortical neurons can give a detailed picture of concurrent voluntary movement (e.g. Georgopoulos et al., 1986, Schwartz, 1993, Chapin et al., 1999, Wessberg et al., 2000). Because these methods are invasive, the threshold for their clinical use would presumably be higher than for methods based on scalp-recorded EEG activity, and they would probably be used mainly by those with extremely severe disabilities. At the same time, they might support more rapid and precise communication and control than the scalp-recorded EEG.

This review summarizes the current state of BCI research with emphasis on its application to the needs of those with severe neuromuscular disabilities. In order to address all current BCI research, it includes approaches that use standard scalp-recorded EEG as well as those that use epidural, subdural, or intracortical recording. While all these present-day BCIs use electrophysiological methods, the basic principles of BCI design and operation discussed here should apply also to BCIs that use other methods to monitor brain activity (e.g. MEG, fMRI). The next sections describe the essential elements of any BCI and the several categories of electrophysiological BCIs, review current research, consider prospects for the future, and discuss the issues most important for further BCI development and application.

Section snippets

Dependent and independent BCIs

A BCI is a communication system in which messages or commands that an individual sends to the external world do not pass through the brain's normal output pathways of peripheral nerves and muscles. For example, in an EEG-based BCI the messages are encoded in EEG activity. A BCI provides its user with an alternative method for acting on the world. BCIs fall into two classes: dependent and independent.

A dependent BCI does not use the brain's normal output pathways to carry the message, but

Present-day BCIs

While many studies have described electrophysiological or other measures of brain function that correlate with concurrent neuromuscular outputs or with intent and might therefore function in a BCI system, relatively few peer-reviewed articles have described human use of systems that satisfy the BCI definition given in Section 2.1 and illustrated in Fig. 1, systems that give the user control over a device and concurrent feedback from the device. These studies are reviewed here. Studies from the

The future of BCI-based communication and control: key issues

Non-muscular communication and control is no longer merely speculation. The studies reviewed in the previous section show that direct communication from the brain to the external world is possible and can serve useful purposes. At the same time, the reality does not yet match the fantasy (e.g. Thomas, 1977): BCIs are not yet able to fly airplanes and are not likely to be doing so anytime soon. Present independent BCIs in their best moments reach 25 bits/min. For those who have no voluntary

Conclusions

A BCI allows a person to communicate with or control the external world without using the brain's normal output pathways of peripheral nerves and muscles. Messages and commands are expressed not by muscle contractions but rather by electrophysiological phenomena such as evoked or spontaneous EEG features (e.g. SCPs, P300, mu/beta rhythms) or cortical neuronal activity. BCI operation depends on the interaction of two adaptive controllers, the user, who must maintain close correlation between his

Acknowledgements

Work in the authors' laboratories has been supported by the National Center for Medical Rehabilitation Research, National Institute of Child Health and Human Development, National Institutes of Health (NIH) in the USA, by the Deutsche Forschungsgemeinschaft (DFG) in Germany, and by the Fonds zur Förderung der wissenschaftlichen Forschung in Austria.

References (198)

  • E. Hardman et al.

    Frontal interhemispheric asymmetry: self regulation and individual differences in humans

    Neurosci Lett

    (1997)
  • A. Hufschmidt et al.

    Statistical evaluation of slow brain potentials on the basis of single trials

    Electroenceph clin Neurophysiol

    (1990)
  • J. Kaiser et al.

    Self-initiation of EEG-based communication in paralyzed patients

    Clin Neurophysiol

    (2001)
  • P.R. Kennedy

    The cone electrode: a long-term electrode that records from neurites grown onto its recording surface

    J Neurosci Methods

    (1989)
  • B. Kotchoubey et al.

    Self-regulation of interhemispheric asymmetry in humans

    Neurosci Lett

    (1996)
  • W.N. Kuhlman

    EEG feedback training of epileptic patients: clinical and electroencephalographic analysis

    Electroenceph clin Neurophysiol

    (1978)
  • W. Lang et al.

    Electric and magnetic fields of the brain accompanying internal simulation of movement

    Cogn Brain Res

    (1996)
  • F.H. Lopes da Silva

    Neural mechanisms underlying brain waves: from neural membranes to networks

    Electroenceph clin Neurophysiol

    (1991)
  • D.J. McFarland et al.

    Spatial filter selection for EEG-based communication

    Electroenceph clin Neurophysiol

    (1997)
  • B.Z. Allison et al.

    ERPs in response to different display parameters and implications for brain–computer interface systems

    Soc Neurosci Abstr

    (2000)
  • E.O. Altenmüller et al.

    Psychophysiology and the EEG

  • P. Anderer et al.

    Artifact processing in computerized analysis of sleep EEG – a review

    Neuropsychobiology

    (1999)
  • C.W. Anderson et al.

    Multivariate autoregressive models for classification of spontaneous electroencephalographic signals during mental tasks

    IEEE Trans Biomed Eng

    (1998)
  • W.W. Armstrong et al.

    Adaptive logic networks

    Handbook of neural computation

    (1996)
  • F. Babiloni et al.

    Linear classification of low-resolution EEG patterns produced by imagined hand movements

    IEEE Trans Rehabil Eng

    (2000)
  • A.B. Barreto et al.

    A practical EMG-based human–computer interface for users with motor disabilities

    J Rehabil Res Dev

    (2000)
  • J.D. Bayliss et al.

    A virtual reality testbed for brain–computer interface research

    IEEE Trans Rehabil Eng

    (2000)
  • H. Berger

    Uber das electrenkephalogramm des menchen

    Arch Psychiatr Nervenkr

    (1929)
  • N. Birbaumer et al.

    Slow potentials of the cerebral cortex and behavior

    Physiol Rev

    (1990)
  • N. Birbaumer

    Slow cortical potentials: their origin, meaning, and clinical use

  • N. Birbaumer et al.

    A spelling device for the paralyzed

    Nature

    (1999)
  • N. Birbaumer

    Rain Man's revelations

    Nature

    (1999)
  • N. Birbaumer et al.

    The thought translation device (TTD) for completely paralyzed patients

    IEEE Trans Rehabil Eng

    (2000)
  • G.E. Birch et al.

    Single-trial processing of event-related potentials using outlier information

    IEEE Trans Biomed Eng

    (1993)
  • G.E. Birch et al.

    Brain–computer interface research at the Neil Squire Foundation

    IEEE Trans Rehabil Eng

    (2000)
  • A.H. Black et al.

    Avoidance training of hippocampal theta waves in flaxedilized dogs and its relation to skeletal movement

    J Comp Physiol Psychol

    (1970)
  • A.H. Black

    The direct control of neural processes by reward and punishment

    Am Sci

    (1971)
  • G.T. Carter

    Rehabilitation management in neuromuscular disease

    J Neurol Rehabil

    (1997)
  • J.K. Chapin et al.

    Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex

    Nat Neurosci

    (1999)
  • G.E. Chatrian

    The mu rhythm

  • Y.L. Chen et al.

    The new design of an infrared-controlled human–computer interface for the disabled

    IEEE Trans Rehabil Eng

    (1999)
  • M.G.H. Coles et al.

    Event-related potentials: an introduction

  • M.D. Craggs

    Cortical control of motor prostheses: using the cord-transected baboon as the primate model for human paraplegia

    Adv Neurol

    (1975)
  • A.J. Dalton

    Discriminative conditioning of hippocampal electrical activity in curarized dogs

    Comm Behav Bio

    (1969)
  • J.L. De Foa et al.

    Issues in cochlear prosthetics from an international survey of opinions

    Int J Technol Assess Health Care

    (1991)
  • A.J. Dewan

    Occipital alpha rhythm, eye position and lens accommodation

    Nature

    (1967)
  • J.P. Donoghue et al.

    Motor areas of the cerebral cortex

    J Clin Neurophysiol

    (1994)
  • E. Donchin

    Presidential address, 1980. Surprise!…Surprise?

    Psychophysiology

    (1981)
  • E. Donchin et al.

    The mental prosthesis: assessing the speed of a P300-based brain–computer interface

    IEEE Trans Rehabil Eng

    (2000)
  • E.V. Evarts

    Pyramidal tract activity associated with a conditioned hand movement in the monkey

    J Neurophysiol

    (1966)
  • Cited by (6430)

    View all citing articles on Scopus
    View full text