System turns imagined handwriting into text

Two implanted electrode arrays record the brain activity produced when thinking about writing letters. This information is collected and processed in real-time by a computer, which converts that data into words on a screen. Shenoy lab & Erika Woodrum (artist)

Advances in technology allow people who have lost the ability to move or speak to communicate again. Brain-computer interfaces, or BCIs, enable direct communication between the brain and an external computer. BCIs decode neural activity and translate it into commands to accomplish the user’s goal. Using BCI technology, people who are paralyzed can control a robotic arm or type with a computer cursor through brain activity.

However, point-and-click typing using BCIs can be a slow process, making it inefficient to use. Researchers are looking for new approaches to speed BCI communication. Focusing on the brain signals used for rapid, fine movements like handwriting might speed the process. A team led by Dr. Francis Willett and Dr. Krishna Shenoy of Stanford developed a BCI that decodes thoughts of handwriting movements and translates them into text in real time. They assessed the speed and accuracy of the system with a person who was paralyzed from a spinal cord injury.

The study was funded in part by NIH’s BRAIN Initiative, National Institute of Neurological Disorders and Stroke (NINDS) and National Institute on Deafness and Other Communication Disorders (NIDCD). Findings were published in Nature on May 12, 2021.

Researchers decoded the neural signals associated with writing letters, then displayed typed versions of those letters in real time. They hope their invention could one day help people with paralysis communicate.

Researchers asked the participant to attempt to write as if his hand was not paralyzed. He was told to imagine he was holding a pen on a piece of ruled paper. He then pictured copying individual letters displayed on the screen, as well as symbols for spaces and stops.

Electrodes implanted in the brain recorded signals from the participant’s motor cortex, the region of the brain that controls fine movement. Even after a person is paralyzed from a spinal cord injury, their motor cortex still works and produces neural signals for the intended movement. The electrodes recorded brain activity from approximately 200 individual neurons that responded collectively to the “writing” of each character. 

These brain signals were then used to train a machine-learning computer algorithm to identify the neural patterns representing individual letters. After a series of training sessions, the system allowed the participant to form new sentences, with the computer displaying letters in real time.

The handwriting BCI system is a marked improvement over existing BCIs that move a cursor to “type” words on a screen. Using the new system, the participant was able to compose sentences and communicate with others at a speed of about 90 characters per minute with 94% accuracy. This speed is comparable to someone of a similar age typing on a smartphone. In contrast, “point-and-click” interfaces have only achieved about 40 characters per minute.

“Attempting to write each letter produces a unique pattern of activity in the brain, making it easier for the computer to identify what is being written with much greater accuracy and speed,” Willett explains.

“This study represents an important milestone in the development of BCIs and machine learning technologies that are unraveling how the human brain controls processes as complex as communication,” says Dr. John Ngai, director of the NIH BRAIN Initiative. “This knowledge is providing a critical foundation for improving the lives of others with neurological injuries and disorders.”

While preliminary, this technology could help those who have lost the ability to write and speak. The clinical trial, called BrainGate2, continues to test this and other BCI technologies.