The human-computer connection
An overview of brain-computer interfaces
This article introduces the field of brain-computer interfaces (BCI), which allows the control of devices without the generation of any active motor output but directly from the decoding of the user’s brain signals. Here we review the current state of the art in the BCI field, discussing the main components of such an interface and illustrating ongoing research questions and prototypes for controlling a large variety of devices, from virtual keyboards for communication to robotics systems to replace lost motor functions and even clinical interventions for motor rehabilitation after a stroke. The article concludes with some insights into the future of BCI.
Keywords: brain-computer interfaces, brain signal processing, machine learning, robotics, rehabilitation.
In a brain-computer interface (BCI), neural signals recorded from the brain are fed into a decoding algorithm that translates these signals into outputs so that people with physical disabilities can control a variety of devices such as virtual keyboards (Birbaumer et al., 1999; Sellers, Ryan, & Hauser, 2014; Vansteensel et al., 2016), games (Perdikis, Tonin, Saeedi, Schneider, & Millán, 2018), arm and hand robots (Collinger et al., 2013; Hochberg et al., 2012) mobile robots (Leeb et al., 2015), and wheelchairs (Carslon & Millán, 2013; Ron-Angevin et al., 2017). For instance, Figure 1 illustrates a brain-controlled wheelchair. Feedback from the prosthetic device, conveyed to the user either via normal sensory pathways or directly through brain stimulation, establishes a closed control loop. BCI technology offers a natural way to augment human capabilities by providing a new interaction link with the outside world. In this respect, it is particularly relevant as an aid for patients with severe neuromuscular disabilities, although it also opens up new possibilities in human-machine interaction for able-bodied people.
«Brain-computer interface technology offers a natural way to augment human capabilities by providing a new interaction link with the outside world»
The central tenet of a BCI is the capability to distinguish between different patterns of brain activity, each being associated with a particular intention or mental command. Hence, adaptation is a key component of a BCI, because, on the one hand, users must learn to modulate their brainwaves so as to generate distinct brain patterns, while, on the other, machine learning techniques ought to discover the individual brain patterns characterizing the mental tasks executed by the user. In essence, a BCI is a two-learner system that must engage in a mutual adaptation process (Carmena, 2013; Perdikis et al., 2018). This process starts by selecting discriminant, stable features – namely, user-specific brain components that maximize the separability between mental commands and that, because of the nonstationary nature of brain signals, are stable over time – to build optimal models to decode the user’s intention. Examples of user-specific brain components that are fed to personalized decoders are the power of some frequency components or the amplitude of band-pass filtered signals1 over certain brain areas. These initial features represent those brain components that the user can naturally modulate and, via feedback received during online BCI training, learn to control voluntarily.
As an example of this mutual learning approach to BCI, a recent study investigated the hypothesis that mutual learning is a critical factor for successful BCI translational applications (Perdikis et al., 2018). Contrary to a popular trend of focusing almost exclusively on the machine learning aspects of BCI training, a holistic mutual learning training approach grounded symmetrically on all three learning pillars (at the machine, subject, and application level) was the optimal training apparatus for preparing two end-user participants for the Cybathlon BCI race in 2016, the first international BCI competition where users with severe disabilities operated a device. In particular, two severely impaired participants, both suffering from chronic spinal cord injury, were trained to control their avatar in a virtual BCI race game. The competition outcomes substantiated the effectiveness of mutual learning as one of them won the gold medal and the two of them established the best three race times (Figure 2). Most importantly, learning correlates could be derived at all levels of the interface – application, BCI output, and EEG neuroimaging – of the two end-users, from sufficiently longitudinal evaluations and, what is more, under real-world and even adverse conditions.
How to record brain signals for BCI
What kind of brain signals can be exploited to directly control devices? Electrical activity is the natural candidate because of its excellent time resolution – changes in brain activity can be monitored at the millisecond range. We can record the electrical brain activity invasively or non-invasively (Figure 3). The former technique employs microelectrode arrays implanted in the brain that record the activity of single neurons – or from small neuronal populations that give rise to local field potentials. The overall concerted activity of neuronal populations can also be recorded invasively with electrodes placed on the surface of the brain, the so-called electrocorticography. Non-invasive BCIs mainly use electroencephalographic activity recorded from electrodes placed on the scalp to measure the synchronous activity of millions of cortical neurons. These signals in local field potentials, electrocorticography, and electroencephalography are similar in nature, although at different spatial levels – microscopic, mesoscopic, and macroscopic, respectively.
Les ICC no invasives utilitzen principalment activitat electroencefalogràfica enregistrada mitjançant elèctrodes ubicats en el cuir cabellut per mesurar l’activitat sincrònica de milions de neurones corticals. La naturalesa dels senyals dels potencials de camp local, l’electrocorticografia i l’electroencefalografia, és semblant, però ocorren a diferents escales espacials –microscòpica, mesoscòpica i macroscòpica, respectivament.
«Endogenous brain-computer interfaces naturally suit applications such as control of robotics devices and motor rehabilitation»
Invasive approaches carry very detailed information about users’ intended actions. However, they damage brain tissue and provide a limited coverage, while motor and decision-making processes involve large brain networks. For humans, however, non-invasive approaches are ideal to bring BCI technology to a large population. Electroencephalography also provides coverage of large cortical areas. But its signals suffer from a reduced spatial resolution and increased noise when measurements are taken on the scalp. In summary, brain recordings at all three levels provide complementary advantages, and a combination of technologies may be necessary in order to achieve the ultimate goal of controlling neuroprostheses capable to replicate any kind of body movement as easily as able-bodied people control their natural limbs (Millán & Carmena, 2010).
Voluntary activity vs evoked potentials
A BCI can exploit two kinds of brain phenomena. One of them is related to signals associated with external sensory stimulation – such as visual flashes or auditory tones –, while the other is connected with endogenous voluntary decision processes – such as the imagination of movements. In the former case, the brain reacts with so-called «evoked potentials». One of such evoked potentials is the P300 (Sellers et al., 2014) which is elicited by an awaited infrequent event that appears at centro-parietal locations along the midline of the scalp, independently of the sensory stimulation modality. As its name indicates, it is a positive wave peaking around 300 milliseconds after task-relevant stimuli. The amplitude of the P300 depends on the frequency of stimulus occurrence – less frequent stimuli produce larger responses – and task relevance. Evoked potentials are, in principle, easy to pick up. Nevertheless, the necessity of external stimulation severely limits the applicability of evoked potentials to tasks requiring continuous control such as in robotics.
In the case of endogenous BCIs, users can voluntarily modulate brain activity at different frequency ranges – or rhythms. Populations of neurons can form complex networks with feedback loops, which give rise to oscillatory activity. In general, the frequency of such oscillations becomes slower the larger the size of the synchronized neuronal assemblies. A particularly relevant rhythm can be recorded from the central region of the scalp overlying the sensorimotor cortex during the imagination of body movements. Correlates of imaginary movements can be recorded at any scale: microscopic (single unit activity and local field points), mesoscopic (electrocorticography), and macroscopic (electroencephalography). Apart from their different degrees of spatial resolution, microscopic and mesoscopic signals also have a broader bandwidth (up to 300–500 Hz) than macroscopic signals (normally, less than 100 Hz). Endogenous BCIs naturally suit applications such as control of robotics devices and motor rehabilitation.
BCIs at work
Researchers expand a BCI, especially those based on electroencephalography, with a few principles to support robust control of devices. The most important one is «shared control» or context awareness, demonstrated in a large variety of devices such as mobile robots, wheelchairs, telepresence robots, lower limb exoskeletons, virtual keyboards, and games. In a shared autonomy framework, the outputs of the BCI are combined with the information about the environment (e.g., obstacles perceived by the robot sensors in the case of a wheelchair, or written letters in the case of a virtual keyboard) and the robot itself (position and velocities) to better estimate the user’s intent, or even override the mental commands in critical situations (Carslon & Millán, 2013). Shared control is not only an efficient engineering solution, but it is also rooted in the fact that human motor control results from the combined activity of the cerebral cortex, subcortical areas, and spinal cord. In fact, many elements of skilled movements, from manipulation to walking, are mainly handled at the brainstem and spinal cord level with cortical areas mainly providing an abstraction of the desired command. This organization supports the hypothesis that complex tasks can be achieved using the low-dimensional output of a BCI, provided the BCI is coupled to an intelligent robotic device that executes the detailed sequence of low-level commands mimicking the role of subcortical and spinal cord levels in human motor control. As a result, shared control also reduces users’ cognitive workload.
A brain-controlled wheelchair (Carslon & Millán, 2013; Ron-Angevin et al., 2017) (Figure 1) illustrates the future of intelligent neuroprostheses that, like our spinal cord and musculoskeletal system, work in tandem with motor commands decoded from the user’s brain cortex. Users can drive it reliably and safely over long periods of time thanks to the incorporation of shared control techniques. This relieves users from the need to continuously deliver all the necessary low-level control parameters and, so, reduces their cognitive workload and facilitates split attention between different tasks like driving the chair and checking the environment.
«The users of a BCI must learn to modulate their brainwaves so as to generate distinct brain patterns»
Another principle used to increase the robustness of a BCI is to decode, and integrate into the neuroprosthetic control loop, neural correlates of the user’s «cognitive perceptual processes» resulting from actions executed by the brain-controlled device and that are crucial for volitional interaction. A major example of such correlates is awareness to errors made by the BCI in decoding the user’s intention (Chavarriaga, Sobolewski, & Millán, 2014). Detection of these error-related potentials, indicating when the user perceives BCI errors, can be then used to correct and improve the system’s performance.
A further component that facilitates intuitive and natural control of motor neuroprosthetics is the incorporation of rich multimodal feedback and neural correlates of perceptual processes resulting from this feedback. Realistic sensory feedback must convey artificial tactile and proprioceptive information – i.e., the awareness of the position and movement – of the neuroprosthesis. This type of sensory information has potential to significantly improve the control of the prosthesis by allowing the user to feel the environment in cases in which natural sensory afferents are compromised – either through other senses or by stimulating the body or even the nervous system directly to recover the lost sensation (Raspopovic et al., 2014). Furthermore, rich multimodal feedback is essential to promote the user’s agency and ownership of the prosthesis.
BCIs for motor rehabilitation after a stroke
In addition to motor substitution, where a BCI bypasses a central nervous injury to control a neuroprosthesis, BCI technology can also facilitate motor rehabilitation (Figure 4). This is an emerging area of research and application, especially for stroke. The rationale for BCI-based motor rehabilitation is twofold. Firstly, and perhaps more pragmatically, contrary to most other well-established rehabilitation paradigms requiring some degree of residual mobility, BCI technology can assist stroke patients even in the absence of residual motor activity. Secondly, and most importantly, it can promote neuroplasticity so that lost functions due to stroke can be recoded in perilesional cortical areas.
A BCI can boost beneficial functional activity-dependent plasticity provided it delivers rich somatosensory and proprioceptive feedback contingent to suitable motor-related cortical activity associated with the movement attempt of the paralyzed limb. A recent study shows that BCI coupled to functional electrical stimulation (FES) elicits significant, clinically relevant, and lasting motor recovery of arm and hand function in chronic stroke survivors more effectively than sham FES (Biasiucci et al., 2018). Such recovery is associated with quantitative signatures of functional neuroplasticity. BCI patients exhibited a significant functional recovery after the intervention which remained six-to-twelve months after the end of therapy. As an example, two patients in the BCI group had a complete hand paralysis and, in one case, the patient participated in the BCI-FES therapy fifteen years after a stroke, for whom a recovery of hand activity is exceptional. Both patients regained voluntary muscular contraction resulting in wrist extension and signs of fingers extension. On the contrary, none of the plegic sham-FES patients showed signs of recovery.
Furthermore, electroencephalography analysis pinpointed significant differences in favor of the BCI group, mainly consisting in an increase in functional connectivity between motor areas in the affected hemisphere. This increase was significantly correlated with functional improvement. Altogether, these results illustrate how a BCI-FES therapy can drive significant functional recovery and purposeful plasticity thanks to the contingent activation of the body natural efferent (motor-related brain activity) and afferent (somatosensory and proprioceptive feedback) pathways.
Current BCI technology, in particular electroencephalography-based, enables the operation of relatively simple brain-actuated devices. No doubt, this represents an important achievement for motor-disabled people. Yet, robust and natural brain interaction with more complex devices remains a major challenge. A related issue is to demonstrate the benefit of BCI for disabled people outside laboratory conditions. Only a few studies have been conducted up to now (Perdikis et al., 2018; Vansteensel et al., 2016). In parallel, as the BCI field is entering a more mature phase of development, time is ripe to design new interaction modalities for able-bodied people. The idea is not to directly control a device via the BCI, but to enhance the interaction experience by predicting actions the user will do (or not do) as well as decoding the user’s cognitive state. This will allow the intelligent device to assist the user in the most convenient way, thus achieving a seamless personalized interaction. An example of this emerging research avenue is the use of BCIs for enhancing the car driving experience (Chavarriaga et al., 2018).
«Robust and natural brain interaction with more complex devices remains a major challenge»
Finally, such future BCI systems will require better recording technology for both invasive and non-invasive brain signals. In the former case, one area of active research is the design of safe biophysical interfaces that, in addition, should be ultra-low power and wireless. In the latter case, examples of new technology are dry electrodes that do not require any gel and can be integrated into aesthetic helmets and skin sensors that can remain operational for months.
1 A band-pass filter passes frequencies within a particular range and rejects the rest. The resulting signal is known as a band-pass filtered signal. (Go back)
Biasiucci, A., Leeb, R., Iturrate, I., Perdikis, S., Al-Khodairy, A., Corbet, T. A., … Millán, J. d. R. (2018). Brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. Nature Communications, 9, 2421. doi: 10.1038/s41467-018-04673-z
Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., … Flor, H. (1999). A spelling device for the paralysed. Nature, 398(6725), 297–298. doi: 10.1038/18581
Carslon, T., & Millán, J. d. R. (2013). Brain-controlled wheelchairs: A robotic architecture. IEEE Robotics and Automation Magazine, 20(1), 65–73. doi: 10.1109/MRA.2012.2229936
Carmena, J. M. (2013). Advances in neuroprosthetic learning and control. PLOS Biology, 11(5), e1001561. doi: 10.1371/journal.pbio.1001561
Chavarriaga, R., Sobolewski, A., & Millán, J. d. R. (2014). Errare machinale est: The use of error-related potentials in brain-machine interfaces. Frontiers in Neuroscience, 8, 208. doi: 10.3389/fnins.2014.00208
Chavarriaga, R., Ušćumlić, M., Zhang, H., Khaliliardali, Z., Aydarkhanov, R., Saeedi, S., … Millán, J. d. R. (2018). Decoding neural correlates of cognitive states to enhance driving experience. IEEE Transactions on Emerging Topics in Computational Intelligence, 2(4), 288–297. doi: 10.1109/TETCI.2018.2848289
Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., … Schwartz, A. B. (2013). High-performance neuroprosthetic control by an individual with tetraplegia. Lancet, 381, 557–564. doi: 10.1016/S0140-6736(12)61816-9
Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse N. Y., Simeral, J. D., Vogel, J., & Donoghue, J. P. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature, 485, 372–375. doi: 10.1038/nature11076
Leeb, R., Tonin, L., Rohm, M., Desideri, L., Carlson, T., & Millán, J. d. R. (2015). Towards independence: A BCI telepresence robot for people with severe motor disabilities. Proceedings of the IEEE, 103(6), 969–982. doi: 10.1109/JPROC.2015.2419736
Millán, J. d. R., & Carmena, J. M. (2010). Invasive or noninvasive: Understanding brain-machine interface technology. IEEE Engineering in Medicine and Biology Magazine, 29(1), 16–22. doi: 10.1109/MEMB.2009.935475
Perdikis, S., Tonin, L., Saeedi, S., Schneider, C., & Millán, J. d. R. (2018). The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users. PLOS Biology, 16(25), e2003787. doi: 10.1371/journal.pbio.2003787
Raspopovic, S., Capogrosso, M., Petrini, F. M., Bonizzato, M., Rigosa, J., Di Pino, G., … Micera, S. (2014). Restoring natural sensory feedback in real-time bidirectional hand prostheses. Science Translational Medicine, 6(222), 222ra19. doi: 10.1126/scitranslmed.3006820
Ron-Angevin, R., Velasco-Álvarez, F., Fernández-Rodríguez, A., Díaz-Estrella, A., Blanca-Mena, M. J., & Vizcaíno-Martín, F. J. (2017). Brain-computer interface application: Auditory serial interface to control a two-class motor-imagery-based wheelchair. Journal of NeuroEngineering and Rehabilitation, 14(1), 49. doi: 10.1186/s12984-017-0261-y
Sellers, E. W., Ryan, D. B., & Hauser, C. K. (2014). Noninvasive brain-computer interface enables communication after brainstem stroke. Science Translational Medicine, 6(257), 257re7. doi: 10.1126/scitranslmed.3007801
Vansteensel, M. J., Pels, E. G. M., Bleichner, M. G., Branco, M. P., Denison, T., Freudenburg, Z. V., … Ramsey, N. F. (2016). Fully implanted brain-computer interface in a locked-in patient with ALS. New England Journal of Medicine, 375, 2060–2066. doi: 10.1056/NEJMoa1608085