Scientists Develop AI That Speaks for You

For those who have suffered a stroke or disease that impacts how the brain transmits speech signals, the inability to properly communicate can be hugely frustrating. In many cases, the brain still processes the signals needed for speech. However, they get lost or stuck in the brain, meaning the words can’t be formed by the mouth or vocal cords.

There are a number of technologies out there that enable those who have lost the ability to speak to communicate using their eyes or other small movements, controlling a cursor or selecting on-screen letters. More recently, three research teams have made progress towards transforming data from electrodes in the brain directly into computer-generated speech.

The challenge of neural network research

Conducting research of this nature is both complex and dangerous, requiring the skull to be opened and electrodes to be implanted in the brain. Much like the measurement of biological nanoparticles, there is a need for extreme precision. This means researchers can only take recordings in rare cases.

One such case is during the removal of a brain tumour, when electrodes are used to help surgeons locate and avoid key speech areas. Alternatively, researchers can take recordings when a person with epilepsy has electrodes implanted for a few days to pinpoint the origin of seizures prior to surgery.

The groups of researchers made the most of the precious data they collected by using neural networks. These networks process complicated patterns using layers of computational ‘nodes’. The networks gather information by adjusting any connections between nodes, which in this case was between recordings of speech that a person produces or hears and the brain activity data during that time. 

Study findings

The first group of researchers, led by Nima Mesgarani, a computer scientist at Columbia University, looked at data from five people with epilepsy. Their neural network analysed recordings taken from the auditory cortex, which is active during both speech and listening, as patients listened to recordings of stories or people naming digits from 0-9. The computer reconstructed spoken numbers solely from neural data, with 75% accuracy.

The second research team, led by computer scientist, Tanja Schultz, analysed data from 6 people undergoing surgery to remove brain tumours. As the team recorded patients reading single-syllable words aloud, electrodes recorded the brain’s speech planning and motor areas. Their neural network then mapped the electrode readings to the audio clips and reconstructed words from brain data. Around 40% of the computer-generated words were understandable.

Finally, Edward Chang, a neurosurgeon from the University of California, attempted to reconstruct entire sentences from the brain data of three epilepsy patients as they read aloud. Then, in an online test, 166 participants listened to one of the sentences before selecting the correct option from 10 written choices. Sentences were correctly identified more than 80% of the time.

While nobody has perfected the technology yet, it might not be long before science has a solution for people suffering from aphasia – or language loss.