Note: Single-source report; awaiting corroboration.
A National Institutes of Health (NIH)-funded team at the University of California, San Francisco (UCSF) has demonstrated a brain-computer interface (BCI) that allows individuals with vocal tract and bodily paralysis to communicate through simultaneous speech and upper-body gestures. This is the first BCI system reported to enable both communication modes at once, closely replicating natural expression.
Researchers used machine learning to decode unique brain activity corresponding to combined speech and physical gestures in three patients. Two participants controlled a personalized full-body virtual avatar in real time through the BCI, translating their brain activity into commands for speech and gestures.
The study focused on patients with paralysis due to conditions such as amyotrophic lateral sclerosis (ALS) and brainstem strokes, which severely impair both verbal and non-verbal communication. Current eye-tracking aids allow text-to-speech, but are limited in speed and expression, and can be physically taxing.
The researchers implanted thin electrocorticography (ECoG) sensor arrays on the participants' motor cortex. Previous UCSF work translated brain signals into digital facial expressions, but this study extended the approach to upper limb gestures using a virtual full-body avatar. Participants performed or attempted to verbalize phrases and gestures, such as hand waves or thumbs-up signs, both separately and simultaneously, enabling data collection for decoding.
According to the study's corresponding author, Edward Chang, M.D., professor of neurological surgery at UCSF, the results provide proof-of-concept that BCIs can restore some freedom and flexibility in communication by engaging the whole motor cortex in a multilayered, dynamic process.