The effect of multimodal and enriched feedback on SMR-BCI performance
Introduction
Brain–computer interfaces (BCIs) based on the modulation of sensorimotor rhythms (SMR) classify differences in the electroencephalogram (EEG) elicited by different motor imagery (MI), actual movement or movement preparation (Pfurtscheller et al., 1997) and translate these into control commands, e.g., for a spelling application (Kübler et al., 2001, Perdikis et al., 2014, Wolpaw et al., 2002) or cursor control on a computer screen (Wolpaw et al., 1991). This provides an alternative communication channel for people diagnosed with neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS), who have only residual control of few muscles, which may be unreliable (Kübler et al., 2005). A limiting factor for the use of a traditional SMR based BCI is that vision must not be compromised in the end-user. For instance, several studies showed that in the last stage of ALS (i.e., completely locked-in) the visual sensory channel cannot be used as a reliable BCI input (De Massari et al., 2013, Murguialday et al., 2011). Sensorimotor rhythms (SMR) refer to localized sinusoidal frequencies in the upper alpha band (10–12 Hz) usually accompanied by changes in synchronization in the beta band (13–25 Hz) (Pfurtscheller and Neuper, 2001), which can be recorded over primary somatosensory and motor cortical areas. SMR decreases or desynchronizes (event related desynchronisation, ERD) with movement or movement imagery in the contralateral sensorimotor areas (Halder et al., 2011, Lotze et al., 1999, Neuper et al., 2005, Pfurtscheller and Aranibar, 1979, Schnitzler et al., 1997). Motor imagery is defined as the mental simulation of a kinesthetic movement (Decety and Inqvar, 1990, Neuper et al., 2005). Signal processing algorithms, individual users’ characteristics, such as psychosocial and physiological parameters (e.g., fine motor skills) or brain structures, can predict performances for SMR based BCIs (Blankertz et al., 2010, Halder et al., 2011; Hammer et al., 2011, Randolph, 2012). Besides these factors, feedback is a necessary feature for initial learning of the BCI skill (Brown, 1970, Kuhlmann, 1978, McFarland et al., 1998, Wolpaw et al., 1991, Wolpaw et al., 2002). The end-user have to be properly trained to be able to successfully control their EEG signals, especially for the use of a BCI based on the recognition of mental imagery tasks (e.g., motor imagery, Neuper and Pfurtscheller, 2010).
To learn modulating SMR power, usually unimodal visual feedback is provided: The end-user receives feedback by an extending bar or a moving cursor (Fig. 1) in one or two dimensions according to the classification results (Schreuder et al., 2010; Neuper and Pfurtscheller, 2010). It provides no information about the quality of the mental imagery as it only gives feedback about which MI is classified at any one point in time. This presentation can be inaccurate, because often the input signal contains a degree of uncertainty that can make a precise classification difficult (Hattie and Timperley, 2007; van Beers et al., 2002). The crucial step is to extract robustly the relevant information from EEG signals in the presence of various noise sources, signal non-stationarity and with limited amount of data available (McFarland and Wolpaw, 2011, van Erp et al., 2012) and to give meaningful and precise feedback (Hattie and Timperley, 2007). Uncertainty is not static and can vary substantially over time. Therefore, we created the visually enriched “funnel feedback” to provide more information about the quality of the EEG signal: we implemented a liquid cursor model in a funnel shape that can provide the end-user with additional information about their input signal. The stability of the EEG was mirrored by the speed of the liquid cursor through the funnel (Fig. 2). Being not in control of a BCI can make its use frustrating (Holz et al., 2015). Frustration has been experienced as problematic in BCI use (Curran and Stokes, 2003) and further Kleih et al. (2010) and Kleih and Kübler (2013) showed that learning an SMR-BCI task is facilitated by increased motivation. If the enriched funnel feedback allowed for better learning, frustration may be lowered and motivation increased.
Although the most common feedback is visual, there is evidence that training can be enhanced by providing multimodal feedback with the same granularity and specificity for each modality (Lotte et al., 2013). Kaufmann et al. (2011) provided their BCI users with a cursor indicating the integrated classifier output, and the instantaneous sign and absolute value, coded as the color and intensity of the cursor. Results suggested that end-user can deal with a multi-dimensional feedback although no significant increase in performance was found. Auditory feedback provides an alternative to a visually-based BCI system (McCreadie et al., 2012, Simon et al., 2014), specifically for those potential end-user with impaired vision. Nijboer et al. (2008) found that although the initial BCI performance in the visual feedback group was superior to the auditory feedback group, there was no significant difference in performance at the end of training. A study by Schreuder et al. (2010) showed that the combination of audio and visual feedback did not lead to an enhancement in BCI performance, whereas Gargiulo et al. (2012) concluded, that multimodal feedback could increase performance in some naïve subjects and could relieve the sense of frustration due to the feeling of not being in control of the visual cue. Thus, studies provided mixed results and further investigation is warranted to elucidate the effect of multimodal feedback on SMR-BCI performance.
The goal of this study was to investigate the effects of a multimodal and visually enriched feedback during SMR based BCI control in a between subject design. For this purpose three end-user groups tested three different forms of feedback: conventional unimodal (visual) cursorbar feedback (CB), unimodal (visual) funnel feedback (UF) and multimodal (visual-auditory) funnel feedback (MF) during five training sessions. All end-users had to perform the same left and right hand motor imagery tasks to control the different type of cursor to the left or right side on the screen. The focus of this study was to investigate how feedback can support end-users in learning to control the BCI therefore we abstained from using communicative characteristics, such as “yes/no” to keep the task as simple as possible. We hypothesized that the enriched visual feedback in combination with auditory feedback would facilitate the learning process, lead to better performance and diminish the level of frustration. The presentation of uncertainty information may render end-users confident toward the functionality of the SMR-BCI, especially during the training phase, where the subject tends to explore different mental strategies to determine the optimal one for achieving control. We further predicted, that the multimodal approach would motivate the end-user.
Section snippets
Participants
Thirty healthy SMR-BCI novices took part in the study which was approved by the Ethical Review Board of the Medical Faculty, University of Tübingen. Each participant was informed about the purpose of the study and signed informed consent prior to participation. None of the participants was excluded from analysis. Of the 30 participants 20 were women, and mean age of the sample was 27.73 years (SD 6.57, range 19–51); six were left-handed.
Experimental set-up
The participants were seated in a comfortable chair
Performance
Feedback accuracy varied largely between participants (mean 62.29 ± 16.1%), covering the full range from chance-level performance (63%) to perfect control (100%). For most participants, performance also varied strongly between sessions. More specifically, the intra-participant performance variability between the five training sessions ranged from 3.5% to 21.3% (mean 6.2 ± 4.4%, Fig. 3). Above chance level performance (>63% hits) was reached by the end-users in 21 training sessions (42%) in the MF
Discussion
We investigated the SMR-BCI performance as a function of feedback type. Performance was measured as the percentage of correct responses during motor imagery tasks. Averaged for all feedback groups 56% of the end-user performed at least one session above chance level with more than 63% correct responses and could, thus, achieve significant control over the required brain response.
During the initial training session significant better performance was measurable in the MF and UF groups as compared
Conclusions
Taken together, healthy participants were able to control a BCI when presented with multimodal funnel feedback of SMR including information about uncertainty. The enriched visual feedback in combination with auditory feedback lead to a significantly better performance in the initial training session. Such feedback may boost initial performance, but beneficial effects were not maintained. Studies possibly with more training sessions are required to replicate this finding and to elucidate the
Acknowledgments
This work is partly supported by the GK Emotion (Research Training Group RTG 1253/2) and the Graduate School of Life Sciences of the University of Würzburg and the EU grant FP7-224631 “TOBI” (Tools for Brain–Computer Interaction) project. This paper only reflects the authors’ views and funding agencies are not liable for any use that may be made of the information contained herein.
Conflict of interest: The authors (T. Sollfrank, A. Ramsay, S. Perdikis, J. Williamson, R. Murray-Smith, R. Leeb,
References (53)
- et al.
The non-invasive Berlin brain–computer interface: fast acquisition of effective performance in untrained subjects
NeuroImage
(2007) - et al.
Neurophysiological predictor of SMR-based BCI performance
NeuroImage
(2010) - et al.
Learning to control brain activity: a review of the production and control of EEG components for driving brain–computer interface (BCI) systems
Brain Cogn
(2003) - et al.
Brain structures participating in mental simulation of motor behavior: a neuropsychological interpretation
Acta Psychol
(1990) - et al.
Acting performance and flow state enhanced with sensory-motor rhythm neurofeedback comparing ecologically valid immersive VR and training screen scenarios
Neurosci Lett
(2010) - et al.
Neural mechanisms of brain–computer interface control
NeuroImage
(2011) - et al.
Motivation modulates the P300 amplitude during brain–computer interface use
Clin Neurophysiol
(2010) - et al.
Brain–computer interfaces and communication in palysis: extinction of goal directed thinking in completely paralysed patients?
Clin Neurophysiol
(2008) - et al.
Brain–computer communication: self-regulation of slow cortical potentials for verbal communication
Arch Phys Med Rehab
(2001) - et al.
Transferring brain–computer interfaces beyond the laboratory: successful application control for motor-disabled users
Artif Intell Med
(2013)
Transition from the locked in to the completely locked-in state: a physiological analysis
Clin Neurophysiol
Imagery of motor actions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG
Cogn Brain Res
An auditory brain–computer interface (BCI)
J Neurosci Methods
Evaluation of event-related desynchronization preceding and following voluntary self-paced movement
Electroencephalogr Clin Neurophysiol
Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks
NeuroImage
EEG-based discrimination between imagination of right and left hand movement
Electroencephalogr Clin Neurophysiol
Classification of epileptiform EEG using a hybrid system based on decision tree classifier and fast Fourier transform
Appl Maths Comput
Involvement of primary motor cortex in motor imagery: a neuromagnetic study
NeuroImage
Brain–computer interfaces for communication and control
Clin Neurophysiol
An EEG-based brain–computer interface for cursor control
Electroencephalogr Clin Neurophysiol
Towards a cure for BCI illiteracy
Brain Topogr
Recognition of aspects of consciousness through association with EEG alpha activity represented by light signal
Psychophysiology
Brain communication in the locked-in state
Brain
Investigating the role of combined acoustic-visual feedback in one-dimensional synchronous brain computer interfaces, a preliminary study
Med Devices
How many people are able to operate an EEG-based brain–computer interface (BCI)?
IEEE T Neu Sys Reh
Cited by (48)
New approaches to clinical electroencephalography analysis in typically developing children and children with autism
2023, Cognitive Systems ResearchCitation Excerpt :In resting wakefulness with eyes closed, similarly to healthy adults, children's EEG typically demonstrated the posteriorly dominant alpha rhythm (8–12 Hz) or theta-rhythm (5–8 Hz) in children younger 8 years. In open eyes, the resting-state EEG usually contains desynchronized oscillations with a relatively low-amplitude and mixed-frequency with sensorimotor rhythm (SMR) over central and frontal areas with the similar frequency and amplitude and wave morphology as posterior dominant alpha rhythm (Louis, et al., 2016) (Nayak & Anilkumar, 2019) (Yuan & He, 2014) (Sollfrank, et al., 2016). Features of posterior dominant alpha and SMRs (both usually described as alpha rhythm) are considered as the most important parameters of the clinical EEG, which help to assess the development of children from infancy into adulthood (Bhavnani, et al., 2021) (Bell, 1998).
Visual and haptic feedback in detecting motor imagery within a wearable brain–computer interface
2023, Measurement: Journal of the International Measurement ConfederationCitation Excerpt :Moreover, it has the potential to enhance cortical activation and system performance as well as increase the pertinence of provided feedback [22–24]. Multimodal feedback is also sought to enhance user engagement [25,26]. The most commonly investigated multimodal feedback combines visual and haptic somatosensory feedback modalities.
Experimenters' Influence on Mental-Imagery based Brain-Computer Interface User Training
2021, International Journal of Human Computer StudiesCitation Excerpt :It is a fundamental component of the MI-BCI training Lotte et al. (2013). Several research were led in order to improve the feedback Pillette (2019), for instance by using more realistic cues Ono et al. (2013); Sollfrank et al. (2016). Users could need specific feedback characteristics depending on their profile Pillette (2019).
Personalized motor imagery prediction model based on individual difference of ERP
2024, Journal of Neural EngineeringA transfer learning-based feedback training motivates the performance of SMR-BCI
2023, Journal of Neural EngineeringBrain–Computer Interfaces for Upper Limb Motor Recovery after Stroke: Current Status and Development Prospects (Review)
2023, Sovremennye Tehnologii v Medicine
- 1
Address: Room SAW320, Computing Science, 17 Lilybank Gardens, Glasgow G12 8QQ, Scotland, United Kingdom.
- 2
Address: EPFL STI IBI-STI CNBI, ELB 118 (Bâtiment ELB), Station 11 CH-1015 Lausanne, Switzerland.
- 3
Address: Room 209 Level 2, Eng-Micro & Nanotechnology, 74 Oakfield Avenue, Glasgow G12 8LP, Scotland, United Kingdom.
- 4
Address: School of Computing Science, Glasgow University, Glasgow G12 8QQ, Scotland, United Kingdom.
- 5
Address: CRR E1 B1 (Clinique Romande de Réadaptation), Av. Grand-Champsec 90, CP 352, CH-1951 Sion, Switzerland.
- 6
Address: EPFL STI IBI-STI CNBI, ELB 138 (Bâtiment ELB), Station 11, CH-1015 Lausanne, Switzerland.
- 7
Address: Raum 302, Psychology I, Marcusstrasse 9-11, 97070 Würzburg, Germany.