A group of experts fromUC San Francisco has successfully developed a “speech neuroprosthesisWhich allowed a man with severe paralysis to communicate using sentences, by translating signals from his brain to the vocal tract directly into words that appear as text on a screen.
The result, which was developed in collaboration with the first participant in a clinical research study, is based on over a decade of efforts by the UCSF neurosurgeon Edward Chang, MD, to develop a speech neuroprosthesis that allows people with paralysis to communicate even if they are unable to speak for themselves.
The study was published on the New England Journal of Medicine.
Speech neuroprosthesis: this is how it works
“As far as we know, this is the first successful demonstration of direct decoding of complete words from the brain activity of someone who is paralyzed and cannot speak.“Explained Chang. Jeanne Robertson and senior author of the study, added: “It shows strong promise of restoring communication by tapping into the brain’s natural language mechanism ”.
Every year, thousands of people lose the ability to communicate due to stroke, accidents or illnesses. With further development, the approach described in this study may one day allow these people to communicate fully thanks to a speech neuroprosthesis.
In previous research, work in the field of communication neuroprosthesis has focused on restoring communication through spelling-based approaches to typing letters one by one in text. Chang’s study differs from these efforts critically: his collaborators stand translating signals intended to control the muscles of the speech system to pronounce words, rather than signals to move the arm or hand to allow typing. Chang said this approach draws on the natural, fluid aspects of speech and promises faster, more organic communication.
“With voice, we normally communicate information at a very high speed, up to 150 or 200 words per minute“, he said, noting that spelling-based approaches using typing, writing, and controlling a cursor are considerably slower and more labor-intensive. “Going straight to the words, as we are doing here, has great advantages because it is closer to how we speak normally ”.
Over the past decade, Chang’s progress has been facilitated by UCSF Epilepsy Center patients undergoing neurosurgery to pinpoint the origins of their seizures, using electrode arrays placed on the surface of the brain. These subjects, all with normal speech, volunteered to have their brain recordings analyzed for language-related activity. Early successes with these volunteer patients paved the way for the current study of people with paralysis and the development of a neuroprosthesis.
Previously, Chang and colleagues fromUCSF Weill Institute for Neurosciences they mapped the cortical activity patterns associated with the vocal tract movements that produce each consonant and vowel. To translate these results into full word speech recognition, David Moses, Ph.D., a postdoctoral engineer in the Chang lab and lead author of the new study, has developed new methods for real-time decoding of those models, as well as incorporating statistical language models to improve accuracy.
But their success in decoding speech in participants who were able to speak did not guarantee that the technology would work in a person whose vocal tract is paralyzed. “Our models needed to learn the mapping between complex patterns of brain activity and expected speech”Moses said. “This presents a great challenge when the participant cannot speak.”
Furthermore, the research team did not know whether the brain signals that control the vocal tract would still remain intact for people who have not been able to move their vocal muscles for many years. “The best way to find out if it could work was to try it“Said Moses.
To investigate the potential of this technology in patients with paralysis, Chang collaborated with his colleague Karunesh Ganguly, MD, Ph.D., associate professor of neurology, to launch a study known as “GOOD BOY“(Brain-Computer Interface Restoration of Arm and Voce). The first participant in the trial was a man in his thirties who suffered a devastating brain stroke more than 15 years ago that severely damaged the connection between his brain, vocal tract and limbs. Since her injury, she has had extremely limited head, neck and limb movements and communicates using a pointer attached to a baseball cap to hit letters on a screen.
The participant, who asked to be indicated as BRAVO1, worked with researchers to create a 50-word vocabulary that Chang’s team could recognize from brain activity using advanced computer algorithms. The vocabulary, which includes words like “water“,”family”And“ good ”, was enough to create hundreds of sentences expressing concepts applicable to the daily life of BRAVO1.
For the study, Chang surgically implanted a series of high-density electrodes on the speech motor cortex of BRAVO1. After the participant’s full recovery, his team recorded 22 hours of neural activity in this brain region over 48 sessions and several months. In each session, BRAVO1 attempted to repeat each of the 50 vocabulary words many times as the electrodes recorded the brain’s signals from his vocal cortex.
To translate the patterns of recorded neural activity into specific words, the two co-authors of Moses, Sean Metzger and Jessie Liu, both graduate students in bioengineering at Chang Lab, they used custom neural network models, which are forms of artificial intelligence. When the participant attempted to speak, these networks distinguished subtle patterns in brain activity to detect word attempts and identify which words he was trying to say.
To test their approach, the team first presented BRAVO1 with short phrases constructed from the 50 vocabulary words and asked him to try repeating them several times. As he made his attempts, the words were decoded by his brain activity, one by one, on a screen.
So the team moved on to ask him questions like “How are you today?” is “Do you want some water?As before, BRAVO1’s attempted speech appeared on the screen. “I’m very good” is “I’m not thirsty”.
Chang and Moses found that the system was able to decode words from brain activity at a rate of up to 18 words per minute with an accuracy of up to 93% (median of 75%). Contributing to the success was a linguistic model applied by Moses which implemented a function of “automatic correction“, Similar to that used by consumer texting and voice recognition software.
Moses characterized the first results of the trial as a proof of principle. “We were thrilled to see the accurate decoding of a variety of meaningful sentences “he said. “We have shown that it is indeed possible to facilitate communication in this way and that it has the potential for use in conversational contexts ”.
Looking ahead, Chang and Moses said they will expand the study to include more participants with severe paralysis and communication deficits. The team is currently working on increasing the number of words in the available vocabulary, as well as improving the speed of speech.
Both stated that while the study focused on a single participant and limited vocabulary, such limitations do not diminish the outcome. “This is an important technological milestone for a person who cannot communicate naturally“, Said Moses,”and demonstrates the potential of this approach to give a voice to people with severe paralysis and language loss ”.