The human-computer connection

This article introduces the field of brain-computer interfaces (BCI), which allows the control of devices without the generation of any active motor output but directly from the decoding of the user’s brain signals. Here we review the current state of the art in the BCI field, discussing the main components of such an interface and illustrating ongoing research questions and prototypes for controlling a large variety of devices, from virtual keyboards for communication to robotics systems to replace lost motor functions and even clinical interventions for motor rehabilitation after a stroke. The article concludes with some insights into the future of BCI.

Keywords: brain-computer interfaces, brain signal processing, machine learning, robotics, rehabilitation.

Introduction

In a brain-computer interface (BCI), neural signals recorded from the brain are fed into a decoding algorithm that translates these signals into outputs so that people with physical disabilities can control a variety of devices such as virtual keyboards (Birbaumer et al., 1999; Sellers, Ryan, & Hauser, 2014; Vansteensel et al., 2016), games (Perdikis, Tonin, Saeedi, Schneider, & Millán, 2018), arm and hand robots (Collinger et al., 2013; Hochberg et al., 2012) mobile robots (Leeb et al., 2015), and wheelchairs (Carslon & Millán, 2013; Ron-Angevin et al., 2017). For instance, Figure 1 illustrates a brain-controlled wheelchair. Feedback from the prosthetic device, conveyed to the user either via normal sensory pathways or directly through brain stimulation, establishes a closed control loop. BCI technology offers a natural way to augment human capabilities by providing a new interaction link with the outside world. In this respect, it is particularly relevant as an aid for patients with severe neuromuscular disabilities, although it also opens up new possibilities in human-machine interaction for able-bodied people.

«Brain-computer interface technology offers a natural way to augment human capabilities by providing a new interaction link with the outside world»

The central tenet of a BCI is the capability to distinguish between different patterns of brain activity, each being associated with a particular intention or mental command. Hence, adaptation is a key component of a BCI, because, on the one hand, users must learn to modulate their brainwaves so as to generate distinct brain patterns, while, on the other, machine learning techniques ought to discover the individual brain patterns characterizing the mental tasks executed by the user. In essence, a BCI is a two-learner system that must engage in a mutual adaptation process (Carmena, 2013; Perdikis et al., 2018). This process starts by selecting discriminant, stable features – namely, user-specific brain components that maximize the separability between mental commands and that, because of the nonstationary nature of brain signals, are stable over time – to build optimal models to decode the user’s intention. Examples of user-specific brain components that are fed to personalized decoders are the power of some frequency components or the amplitude of band-pass filtered signals¹ over certain brain areas. These initial features represent those brain components that the user can naturally modulate and, via feedback received during online BCI training, learn to control voluntarily.

As an example of this mutual learning approach to BCI, a recent study investigated the hypothesis that mutual learning is a critical factor for successful BCI translational applications (Perdikis et al., 2018). Contrary to a popular trend of focusing almost exclusively on the machine learning aspects of BCI training, a holistic mutual learning training approach grounded symmetrically on all three learning pillars (at the machine, subject, and application level) was the optimal training apparatus for preparing two end-user participants for the Cybathlon BCI race in 2016, the first international BCI competition where users with severe disabilities operated a device. In particular, two severely impaired participants, both suffering from chronic spinal cord injury, were trained to control their avatar in a virtual BCI race game. The competition outcomes substantiated the effectiveness of mutual learning as one of them won the gold medal and the two of them established the best three race times (Figure 2). Most importantly, learning correlates could be derived at all levels of the interface – application, BCI output, and EEG neuroimaging – of the two end-users, from sufficiently longitudinal evaluations and, what is more, under real-world and even adverse conditions.

How to record brain signals for BCI

What kind of brain signals can be exploited to directly control devices? Electrical activity is the natural candidate because of its excellent time resolution – changes in brain activity can be monitored at the millisecond range. We can record the electrical brain activity invasively or non-invasively (Figure 3). The former technique employs microelectrode arrays implanted in the brain that record the activity of single neurons – or from small neuronal populations that give rise to local field potentials. The overall concerted activity of neuronal populations can also be recorded invasively with electrodes placed on the surface of the brain, the so-called electrocorticography. Non-invasive BCIs mainly use electroencephalographic activity recorded from electrodes placed on the scalp to measure the synchronous activity of millions of cortical neurons. These signals in local field potentials, electrocorticography, and electroencephalography are similar in nature, although at different spatial levels – microscopic, mesoscopic, and macroscopic, respectively.

Figure 2. A brain-computer interface (BCI) is a system with two learners in a process of mutual adaptation. Users must learn to modulate their brain waves to generate different brain patterns, while machine learning techniques should discover the individual brain patterns that characterize the mental tasks performed by their users. A holistic approach that took this mutual learning process into account proved to be the optimal training system to prepare two users who participated in the Cybathlon BCI race in 2016, the first international BCI competition in which users with severe disabilities operate a device. Both participants (in the pictures) were members of the Brain Tweakers team and had sustained serious chronic spinal injuries, and they entered the competition to control their avatar in a virtual BCI racing game. The two of them scored the top three times in the ICC competition and one of them won the gold medal. / Photo: ETH Zürich / Alessandro Della Bella

Les ICC no invasives utilitzen principalment activitat electroencefalogràfica enregistrada mitjançant elèctrodes ubicats en el cuir cabellut per mesurar l’activitat sincrònica de milions de neurones corticals. La naturalesa dels senyals dels potencials de camp local, l’electrocorticografia i l’electroencefalografia, és semblant, però ocorren a diferents escales espacials –microscòpica, mesoscòpica i macroscòpica, respectivament.

«Endogenous brain-computer interfaces naturally suit applications such as control of robotics devices and motor rehabilitation»

Invasive approaches carry very detailed information about users’ intended actions. However, they damage brain tissue and provide a limited coverage, while motor and decision-making processes involve large brain networks. For humans, however, non-invasive approaches are ideal to bring BCI technology to a large population. Electroencephalography also provides coverage of large cortical areas. But its signals suffer from a reduced spatial resolution and increased noise when measurements are taken on the scalp. In summary, brain recordings at all three levels provide complementary advantages, and a combination of technologies may be necessary in order to achieve the ultimate goal of controlling neuroprostheses capable to replicate any kind of body movement as easily as able-bodied people control their natural limbs (Millán & Carmena, 2010).

Voluntary activity vs evoked potentials

A BCI can exploit two kinds of brain phenomena. One of them is related to signals associated with external sensory stimulation – such as visual flashes or auditory tones –, while the other is connected with endogenous voluntary decision processes – such as the imagination of movements. In the former case, the brain reacts with so-called «evoked potentials». One of such evoked potentials is the P300 (Sellers et al., 2014) which is elicited by an awaited infrequent event that appears at centro-parietal locations along the midline of the scalp, independently of the sensory stimulation modality. As its name indicates, it is a positive wave peaking around 300 milliseconds after task-relevant stimuli. The amplitude of the P300 depends on the frequency of stimulus occurrence – less frequent stimuli produce larger responses – and task relevance. Evoked potentials are, in principle, easy to pick up. Nevertheless, the necessity of external stimulation severely limits the applicability of evoked potentials to tasks requiring continuous control such as in robotics.

Figure 3. The electrical activity of the brain can be recorded invasively or non-invasively. One of the invasive techniques is electrocorticography, which involves a surgical procedure to place a number of electrodes like those shown in the picture on the right on the surface of the brain. Non-invasive brain-computer interfaces often use electroencephalography, which records the synchronized activity of millions of cortical neurons using electrodes located on the scalp, as seen in the example of the picture above. / Photo: R. Alex Durbin / US Air Force

In the case of endogenous BCIs, users can voluntarily modulate brain activity at different frequency ranges – or rhythms. Populations of neurons can form complex networks with feedback loops, which give rise to oscillatory activity. In general, the frequency of such oscillations becomes slower the larger the size of the synchronized neuronal assemblies. A particularly relevant rhythm can be recorded from the central region of the scalp overlying the sensorimotor cortex during the imagination of body movements. Correlates of imaginary movements can be recorded at any scale: microscopic (single unit activity and local field points), mesoscopic (electrocorticography), and macroscopic (electroencephalography). Apart from their different degrees of spatial resolution, microscopic and mesoscopic signals also have a broader bandwidth (up to 300–500 Hz) than macroscopic signals (normally, less than 100 Hz). Endogenous BCIs naturally suit applications such as control of robotics devices and motor rehabilitation.

BCIs at work

Researchers expand a BCI, especially those based on electroencephalography, with a few principles to support robust control of devices. The most important one is «shared control» or context awareness, demonstrated in a large variety of devices such as mobile robots, wheelchairs, telepresence robots, lower limb exoskeletons, virtual keyboards, and games. In a shared autonomy framework, the outputs of the BCI are combined with the information about the environment (e.g., obstacles perceived by the robot sensors in the case of a wheelchair, or written letters in the case of a virtual keyboard) and the robot itself (position and velocities) to better estimate the user’s intent, or even override the mental commands in critical situations (Carslon & Millán, 2013). Shared control is not only an efficient engineering solution, but it is also rooted in the fact that human motor control results from the combined activity of the cerebral cortex, subcortical areas, and spinal cord. In fact, many elements of skilled movements, from manipulation to walking, are mainly handled at the brainstem and spinal cord level with cortical areas mainly providing an abstraction of the desired command. This organization supports the hypothesis that complex tasks can be achieved using the low-dimensional output of a BCI, provided the BCI is coupled to an intelligent robotic device that executes the detailed sequence of low-level commands mimicking the role of subcortical and spinal cord levels in human motor control. As a result, shared control also reduces users’ cognitive workload.
A brain-controlled wheelchair (Carslon & Millán, 2013; Ron-Angevin et al., 2017) (Figure 1) illustrates the future of intelligent neuroprostheses that, like our spinal cord and musculoskeletal system, work in tandem with motor commands decoded from the user’s brain cortex. Users can drive it reliably and safely over long periods of time thanks to the incorporation of shared control techniques. This relieves users from the need to continuously deliver all the necessary low-level control parameters and, so, reduces their cognitive workload and facilitates split attention between different tasks like driving the chair and checking the environment.

«The users of a BCI must learn to modulate their brainwaves so as to generate distinct brain patterns»

Another principle used to increase the robustness of a BCI is to decode, and integrate into the neuroprosthetic control loop, neural correlates of the user’s «cognitive perceptual processes» resulting from actions executed by the brain-controlled device and that are crucial for volitional interaction. A major example of such correlates is awareness to errors made by the BCI in decoding the user’s intention (Chavarriaga, Sobolewski, & Millán, 2014). Detection of these error-related potentials, indicating when the user perceives BCI errors, can be then used to correct and improve the system’s performance.

A further component that facilitates intuitive and natural control of motor neuroprosthetics is the incorporation of rich multimodal feedback and neural correlates of perceptual processes resulting from this feedback. Realistic sensory feedback must convey artificial tactile and proprioceptive information – i.e., the awareness of the position and movement – of the neuroprosthesis. This type of sensory information has potential to significantly improve the control of the prosthesis by allowing the user to feel the environment in cases in which natural sensory afferents are compromised – either through other senses or by stimulating the body or even the nervous system directly to recover the lost sensation (Raspopovic et al., 2014). Furthermore, rich multimodal feedback is essential to promote the user’s agency and ownership of the prosthesis.

BCIs for motor rehabilitation after a stroke

In addition to motor substitution, where a BCI bypasses a central nervous injury to control a neuroprosthesis, BCI technology can also facilitate motor rehabilitation (Figure 4). This is an emerging area of research and application, especially for stroke. The rationale for BCI-based motor rehabilitation is twofold. Firstly, and perhaps more pragmatically, contrary to most other well-established rehabilitation paradigms requiring some degree of residual mobility, BCI technology can assist stroke patients even in the absence of residual motor activity. Secondly, and most importantly, it can promote neuroplasticity so that lost functions due to stroke can be recoded in perilesional cortical areas.

Figure 4. Brain-computer interface technology can also facilitate motor rehabilitation. This is an emerging area of research and applications, especially after a stroke. In the picture, a patient at the Santa Lucia Foundation Hospital in Rome undergoes neurorehabilitation after suffering a stroke. / Photo: Fondazione Santa Lucia

A BCI can boost beneficial functional activity-dependent plasticity provided it delivers rich somatosensory and proprioceptive feedback contingent to suitable motor-related cortical activity associated with the movement attempt of the paralyzed limb. A recent study shows that BCI coupled to functional electrical stimulation (FES) elicits significant, clinically relevant, and lasting motor recovery of arm and hand function in chronic stroke survivors more effectively than sham FES (Biasiucci et al., 2018). Such recovery is associated with quantitative signatures of functional neuroplasticity. BCI patients exhibited a significant functional recovery after the intervention which remained six-to-twelve months after the end of therapy. As an example, two patients in the BCI group had a complete hand paralysis and, in one case, the patient participated in the BCI-FES therapy fifteen years after a stroke, for whom a recovery of hand activity is exceptional. Both patients regained voluntary muscular contraction resulting in wrist extension and signs of fingers extension. On the contrary, none of the plegic sham-FES patients showed signs of recovery.

Furthermore, electroencephalography analysis pinpointed significant differences in favor of the BCI group, mainly consisting in an increase in functional connectivity between motor areas in the affected hemisphere. This increase was significantly correlated with functional improvement. Altogether, these results illustrate how a BCI-FES therapy can drive significant functional recovery and purposeful plasticity thanks to the contingent activation of the body natural efferent (motor-related brain activity) and afferent (somatosensory and proprioceptive feedback) pathways.

Conclusions

Current BCI technology, in particular electroencephalography-based, enables the operation of relatively simple brain-actuated devices. No doubt, this represents an important achievement for motor-disabled people. Yet, robust and natural brain interaction with more complex devices remains a major challenge. A related issue is to demonstrate the benefit of BCI for disabled people outside laboratory conditions. Only a few studies have been conducted up to now (Perdikis et al., 2018; Vansteensel et al., 2016). In parallel, as the BCI field is entering a more mature phase of development, time is ripe to design new interaction modalities for able-bodied people. The idea is not to directly control a device via the BCI, but to enhance the interaction experience by predicting actions the user will do (or not do) as well as decoding the user’s cognitive state. This will allow the intelligent device to assist the user in the most convenient way, thus achieving a seamless personalized interaction. An example of this emerging research avenue is the use of BCIs for enhancing the car driving experience (Chavarriaga et al., 2018).

«Robust and natural brain interaction with more complex devices remains a major challenge»

Finally, such future BCI systems will require better recording technology for both invasive and non-invasive brain signals. In the former case, one area of active research is the design of safe biophysical interfaces that, in addition, should be ultra-low power and wireless. In the latter case, examples of new technology are dry electrodes that do not require any gel and can be integrated into aesthetic helmets and skin sensors that can remain operational for months.