Elsevier

Clinical Neurophysiology

Volume 127, Issue 1, January 2016, Pages 490-498
Clinical Neurophysiology

The effect of multimodal and enriched feedback on SMR-BCI performance

https://doi.org/10.1016/j.clinph.2015.06.004Get rights and content

Highlights

  • Uncertainty is not static and can vary substantially over time, possibly rendering BCI frustrating for the end-user.

  • The multimodal feedback gives direct continuous feedback about the quality of motor imagery classification.

  • Participants were able to control the BCI with the funnel feedback with better performance during the initial session and less frustration compared to the conventional cursorbar feedback.

Abstract

Objective

This study investigated the effect of multimodal (visual and auditory) continuous feedback with information about the uncertainty of the input signal on motor imagery based BCI performance. A liquid floating through a visualization of a funnel (funnel feedback) provided enriched visual or enriched multimodal feedback.

Methods

In a between subject design 30 healthy SMR-BCI naive participants were provided with either conventional bar feedback (CB), or visual funnel feedback (UF), or multimodal (visual and auditory) funnel feedback (MF). Subjects were required to imagine left and right hand movement and were trained to control the SMR based BCI for five sessions on separate days.

Results

Feedback accuracy varied largely between participants. The MF feedback lead to a significantly better performance in session 1 as compared to the CB feedback and could significantly enhance motivation and minimize frustration in BCI use across the five training sessions.

Conclusion

The present study demonstrates that the BCI funnel feedback allows participants to modulate sensorimotor EEG rhythms. Participants were able to control the BCI with the funnel feedback with better performance during the initial session and less frustration compared to the CB feedback.

Significance

The multimodal funnel feedback provides an alternative to the conventional cursorbar feedback for training subjects to modulate their sensorimotor rhythms.

Introduction

Brain–computer interfaces (BCIs) based on the modulation of sensorimotor rhythms (SMR) classify differences in the electroencephalogram (EEG) elicited by different motor imagery (MI), actual movement or movement preparation (Pfurtscheller et al., 1997) and translate these into control commands, e.g., for a spelling application (Kübler et al., 2001, Perdikis et al., 2014, Wolpaw et al., 2002) or cursor control on a computer screen (Wolpaw et al., 1991). This provides an alternative communication channel for people diagnosed with neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS), who have only residual control of few muscles, which may be unreliable (Kübler et al., 2005). A limiting factor for the use of a traditional SMR based BCI is that vision must not be compromised in the end-user. For instance, several studies showed that in the last stage of ALS (i.e., completely locked-in) the visual sensory channel cannot be used as a reliable BCI input (De Massari et al., 2013, Murguialday et al., 2011). Sensorimotor rhythms (SMR) refer to localized sinusoidal frequencies in the upper alpha band (10–12 Hz) usually accompanied by changes in synchronization in the beta band (13–25 Hz) (Pfurtscheller and Neuper, 2001), which can be recorded over primary somatosensory and motor cortical areas. SMR decreases or desynchronizes (event related desynchronisation, ERD) with movement or movement imagery in the contralateral sensorimotor areas (Halder et al., 2011, Lotze et al., 1999, Neuper et al., 2005, Pfurtscheller and Aranibar, 1979, Schnitzler et al., 1997). Motor imagery is defined as the mental simulation of a kinesthetic movement (Decety and Inqvar, 1990, Neuper et al., 2005). Signal processing algorithms, individual users’ characteristics, such as psychosocial and physiological parameters (e.g., fine motor skills) or brain structures, can predict performances for SMR based BCIs (Blankertz et al., 2010, Halder et al., 2011; Hammer et al., 2011, Randolph, 2012). Besides these factors, feedback is a necessary feature for initial learning of the BCI skill (Brown, 1970, Kuhlmann, 1978, McFarland et al., 1998, Wolpaw et al., 1991, Wolpaw et al., 2002). The end-user have to be properly trained to be able to successfully control their EEG signals, especially for the use of a BCI based on the recognition of mental imagery tasks (e.g., motor imagery, Neuper and Pfurtscheller, 2010).

To learn modulating SMR power, usually unimodal visual feedback is provided: The end-user receives feedback by an extending bar or a moving cursor (Fig. 1) in one or two dimensions according to the classification results (Schreuder et al., 2010; Neuper and Pfurtscheller, 2010). It provides no information about the quality of the mental imagery as it only gives feedback about which MI is classified at any one point in time. This presentation can be inaccurate, because often the input signal contains a degree of uncertainty that can make a precise classification difficult (Hattie and Timperley, 2007; van Beers et al., 2002). The crucial step is to extract robustly the relevant information from EEG signals in the presence of various noise sources, signal non-stationarity and with limited amount of data available (McFarland and Wolpaw, 2011, van Erp et al., 2012) and to give meaningful and precise feedback (Hattie and Timperley, 2007). Uncertainty is not static and can vary substantially over time. Therefore, we created the visually enriched “funnel feedback” to provide more information about the quality of the EEG signal: we implemented a liquid cursor model in a funnel shape that can provide the end-user with additional information about their input signal. The stability of the EEG was mirrored by the speed of the liquid cursor through the funnel (Fig. 2). Being not in control of a BCI can make its use frustrating (Holz et al., 2015). Frustration has been experienced as problematic in BCI use (Curran and Stokes, 2003) and further Kleih et al. (2010) and Kleih and Kübler (2013) showed that learning an SMR-BCI task is facilitated by increased motivation. If the enriched funnel feedback allowed for better learning, frustration may be lowered and motivation increased.

Although the most common feedback is visual, there is evidence that training can be enhanced by providing multimodal feedback with the same granularity and specificity for each modality (Lotte et al., 2013). Kaufmann et al. (2011) provided their BCI users with a cursor indicating the integrated classifier output, and the instantaneous sign and absolute value, coded as the color and intensity of the cursor. Results suggested that end-user can deal with a multi-dimensional feedback although no significant increase in performance was found. Auditory feedback provides an alternative to a visually-based BCI system (McCreadie et al., 2012, Simon et al., 2014), specifically for those potential end-user with impaired vision. Nijboer et al. (2008) found that although the initial BCI performance in the visual feedback group was superior to the auditory feedback group, there was no significant difference in performance at the end of training. A study by Schreuder et al. (2010) showed that the combination of audio and visual feedback did not lead to an enhancement in BCI performance, whereas Gargiulo et al. (2012) concluded, that multimodal feedback could increase performance in some naïve subjects and could relieve the sense of frustration due to the feeling of not being in control of the visual cue. Thus, studies provided mixed results and further investigation is warranted to elucidate the effect of multimodal feedback on SMR-BCI performance.

The goal of this study was to investigate the effects of a multimodal and visually enriched feedback during SMR based BCI control in a between subject design. For this purpose three end-user groups tested three different forms of feedback: conventional unimodal (visual) cursorbar feedback (CB), unimodal (visual) funnel feedback (UF) and multimodal (visual-auditory) funnel feedback (MF) during five training sessions. All end-users had to perform the same left and right hand motor imagery tasks to control the different type of cursor to the left or right side on the screen. The focus of this study was to investigate how feedback can support end-users in learning to control the BCI therefore we abstained from using communicative characteristics, such as “yes/no” to keep the task as simple as possible. We hypothesized that the enriched visual feedback in combination with auditory feedback would facilitate the learning process, lead to better performance and diminish the level of frustration. The presentation of uncertainty information may render end-users confident toward the functionality of the SMR-BCI, especially during the training phase, where the subject tends to explore different mental strategies to determine the optimal one for achieving control. We further predicted, that the multimodal approach would motivate the end-user.

Section snippets

Participants

Thirty healthy SMR-BCI novices took part in the study which was approved by the Ethical Review Board of the Medical Faculty, University of Tübingen. Each participant was informed about the purpose of the study and signed informed consent prior to participation. None of the participants was excluded from analysis. Of the 30 participants 20 were women, and mean age of the sample was 27.73 years (SD 6.57, range 19–51); six were left-handed.

Experimental set-up

The participants were seated in a comfortable chair

Performance

Feedback accuracy varied largely between participants (mean 62.29 ± 16.1%), covering the full range from chance-level performance (63%) to perfect control (100%). For most participants, performance also varied strongly between sessions. More specifically, the intra-participant performance variability between the five training sessions ranged from 3.5% to 21.3% (mean 6.2 ± 4.4%, Fig. 3). Above chance level performance (>63% hits) was reached by the end-users in 21 training sessions (42%) in the MF

Discussion

We investigated the SMR-BCI performance as a function of feedback type. Performance was measured as the percentage of correct responses during motor imagery tasks. Averaged for all feedback groups 56% of the end-user performed at least one session above chance level with more than 63% correct responses and could, thus, achieve significant control over the required brain response.

During the initial training session significant better performance was measurable in the MF and UF groups as compared

Conclusions

Taken together, healthy participants were able to control a BCI when presented with multimodal funnel feedback of SMR including information about uncertainty. The enriched visual feedback in combination with auditory feedback lead to a significantly better performance in the initial training session. Such feedback may boost initial performance, but beneficial effects were not maintained. Studies possibly with more training sessions are required to replicate this finding and to elucidate the

Acknowledgments

This work is partly supported by the GK Emotion (Research Training Group RTG 1253/2) and the Graduate School of Life Sciences of the University of Würzburg and the EU grant FP7-224631 “TOBI” (Tools for Brain–Computer Interaction) project. This paper only reflects the authors’ views and funding agencies are not liable for any use that may be made of the information contained herein.

Conflict of interest: The authors (T. Sollfrank, A. Ramsay, S. Perdikis, J. Williamson, R. Murray-Smith, R. Leeb,

References (53)

  • A.R. Murguialday et al.

    Transition from the locked in to the completely locked-in state: a physiological analysis

    Clin Neurophysiol

    (2011)
  • C. Neuper et al.

    Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG

    Cogn Brain Res

    (2005)
  • F. Nijboer et al.

    An auditory brain–computer interface (BCI)

    J Neurosci Methods

    (2008)
  • G. Pfurtscheller et al.

    Evaluation of event-related desynchronization preceding and following voluntary self-paced movement

    Electroencephalogr Clin Neurophysiol

    (1979)
  • G. Pfurtscheller et al.

    Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks

    NeuroImage

    (2006)
  • G. Pfurtscheller et al.

    EEG-based discrimination between imagination of right and left hand movement

    Electroencephalogr Clin Neurophysiol

    (1997)
  • K. Polat et al.

    Classification of epileptiform EEG using a hybrid system based on decision tree classifier and fast Fourier transform

    Appl Maths Comput

    (2007)
  • A. Schnitzler et al.

    Involvement of primary motor cortex in motor imagery: a neuromagnetic study

    NeuroImage

    (1997)
  • J.R. Wolpaw et al.

    Brain–computer interfaces for communication and control

    Clin Neurophysiol

    (2002)
  • J.R. Wolpaw et al.

    An EEG-based brain–computer interface for cursor control

    Electroencephalogr Clin Neurophysiol

    (1991)
  • Allison B, Neuper C. Could anyone use a BCI? In: Tan DS, Nijholt A, editors. Brain–computer interfaces: human–computer...
  • B. Blankertz et al.

    Towards a cure for BCI illiteracy

    Brain Topogr

    (2010)
  • B. Brown

    Recognition of aspects of consciousness through association with EEG alpha activity represented by light signal

    Psychophysiology

    (1970)
  • D. De Massari et al.

    Brain communication in the locked-in state

    Brain

    (2013)
  • G.D. Gargiulo et al.

    Investigating the role of combined acoustic-visual feedback in one-dimensional synchronous brain computer interfaces, a preliminary study

    Med Devices

    (2012)
  • C. Guger et al.

    How many people are able to operate an EEG-based brain–computer interface (BCI)?

    IEEE T Neu Sys Reh

    (2003)
  • Cited by (48)

    • New approaches to clinical electroencephalography analysis in typically developing children and children with autism

      2023, Cognitive Systems Research
      Citation Excerpt :

      In resting wakefulness with eyes closed, similarly to healthy adults, children's EEG typically demonstrated the posteriorly dominant alpha rhythm (8–12 Hz) or theta-rhythm (5–8 Hz) in children younger 8 years. In open eyes, the resting-state EEG usually contains desynchronized oscillations with a relatively low-amplitude and mixed-frequency with sensorimotor rhythm (SMR) over central and frontal areas with the similar frequency and amplitude and wave morphology as posterior dominant alpha rhythm (Louis, et al., 2016) (Nayak & Anilkumar, 2019) (Yuan & He, 2014) (Sollfrank, et al., 2016). Features of posterior dominant alpha and SMRs (both usually described as alpha rhythm) are considered as the most important parameters of the clinical EEG, which help to assess the development of children from infancy into adulthood (Bhavnani, et al., 2021) (Bell, 1998).

    • Visual and haptic feedback in detecting motor imagery within a wearable brain–computer interface

      2023, Measurement: Journal of the International Measurement Confederation
      Citation Excerpt :

      Moreover, it has the potential to enhance cortical activation and system performance as well as increase the pertinence of provided feedback [22–24]. Multimodal feedback is also sought to enhance user engagement [25,26]. The most commonly investigated multimodal feedback combines visual and haptic somatosensory feedback modalities.

    • Experimenters' Influence on Mental-Imagery based Brain-Computer Interface User Training

      2021, International Journal of Human Computer Studies
      Citation Excerpt :

      It is a fundamental component of the MI-BCI training Lotte et al. (2013). Several research were led in order to improve the feedback Pillette (2019), for instance by using more realistic cues Ono et al. (2013); Sollfrank et al. (2016). Users could need specific feedback characteristics depending on their profile Pillette (2019).

    View all citing articles on Scopus
    1

    Address: Room SAW320, Computing Science, 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland, United Kingdom.

    2

    Address: EPFL STI IBI-STI CNBI, ELB 118 (Bâtiment ELB), Station 11 CH-1015 Lausanne, Switzerland.

    3

    Address: Room 209 Level 2, Eng-Micro & Nanotechnology, 74 Oakfield Avenue, Glasgow G12 8LP, Scotland, United Kingdom.

    4

    Address: School of Computing Science, Glasgow University, Glasgow G12 8QQ, Scotland, United Kingdom.

    5

    Address: CRR E1 B1 (Clinique Romande de Réadaptation), Av. Grand-Champsec 90, CP 352, CH-1951 Sion, Switzerland.

    6

    Address: EPFL STI IBI-STI CNBI, ELB 138 (Bâtiment ELB), Station 11, CH-1015 Lausanne, Switzerland.

    7

    Address: Raum 302, Psychology I, Marcusstrasse 9-11, 97070 Würzburg, Germany.

    View full text