Domestic team realizes brain-computer interface "full spectrum Chinese decoding", a major breakthrough with Top 3 accuracy rate of nearly 100%

In August of this year, two back-to-back "Nature" articles demonstrated the powerful capabilities of brain-computer interfaces in language recovery. Most of the existing language brain-computer interface technologies are built for "English and other alphabetic languages" systems. Research on language brain-computer interface systems for "non-alphabetic systems such as Chinese characters" is still blank.

Recently, Professor Mohammed Sawan’s team at the Advanced Neural Chip Center, Professor Zhang Yue’s team and Professor Zhu Junming’s team from the Natural Language Processing Laboratory jointly released their latest research results. The brain-computer interface full-spectrum Chinese decoding has been realized, which to a certain extent fills the gap in international Chinese decoding brain-computer interface technology.

Picture

Paper address: https://www.biorxiv.org/content/10.1101/2023.11.05.562313v1

This study uses stereotactic electroencephalography (SEEG) to collect neural activity signals in the brain corresponding to the pronunciation process of all Mandarin Chinese characters, and combines deep learning algorithms and language models to achieve full spectrum The decoding of Chinese character pronunciation establishes a Chinese brain-computer interface system covering the pronunciation of all Chinese Mandarin characters, achieving end-to-end output from brain activity to complete Mandarin sentences.

Brain-Computer Interface Decoding Chinese

BCI, or brain-computer interface, is widely recognized as the core of the future intersection and integration of life sciences and information technology. This is a research direction with important social value and strategic significance

Brain-computer interface technology is a new type of information transmission channel designed to connect the human or animal brain with external devices Establish a connection path for information exchange. It allows information to bypass the original muscles and peripheral nerve pathways, thereby achieving connection with the outside world, and to a certain extent, replacing human movement, language and other functions

中文字幕A pictographic and syllabic language with more than 50,000 characters is significantly different from English, which is composed of 26 letters, so this is a huge challenge for existing language brain-computer interface systems.

In order to solve this problem, the research team has conducted an in-depth analysis of the pronunciation rules and characteristics of Chinese in the past three years. Starting from the three elements of initial consonants, tones and finals of Chinese pronunciation syllables, and combining the characteristics of the Pinyin input system, they designed a brand new language brain-computer interface system suitable for Chinese

The research team created a Chinese speech-SEEG database of more than 100 hours. They designed a speech library covering all 407 Chinese Pinyin syllables and Chinese pronunciation characteristics, and simultaneously collected EEG signals

This system builds a prediction model by training an artificial intelligence model, which is used to predict the three elements of Chinese character pronunciation syllables (initial consonants, tones and finals). Finally, the system integrates all predicted elements through a language model and combines semantic information to generate the most likely complete Chinese sentence

Picture

The research team evaluated the decoding ability of a brain-computer interface system in daily Chinese environments. In more than one hundred randomly selected decoding tests of complex communication scenarios consisting of two to fifteen characters, the median character error rate averaged 29% across all participants, some decoding by EEG The obtained sentence has a complete accuracy rate of 30%

The relatively efficient decoding performance benefits from the excellent performance of the three independent syllable element decoders and the perfect cooperation of the intelligent language model. In particular, in terms of classifying 21 initial consonants, the accuracy of the initial consonant decoder exceeds 40% (more than 3 times the baseline), and the Top 3 accuracy rate reaches almost 100%; while the tone decoder used to distinguish 4 tones The accuracy rate also reached 50% (more than 2 times the baseline).

The intelligent language model has outstanding performance in the entire language brain-computer interface system. In addition to the important contribution of the three independent syllable element decoders, its powerful automatic error correction capabilities and contextual connection capabilities also played a key role

Picture

This research is a Chinese phonetic language. The BCI decoding research provides a new perspective, and also proves that the performance of the language brain-computer interface system can be significantly improved through powerful language models, providing a new direction for future research on phonetic and written language neuroprostheses

The significance of this work is that it indicates that patients with neurological diseases will soon be able to control computer-generated Chinese sentences through their thoughts, thereby restoring their ability to communicate!

Reference:

https://www.biorxiv.org/content/10.1101/2023.11.05.562313v1

The above is the detailed content of Domestic team realizes brain-computer interface "full spectrum Chinese decoding", a major breakthrough with Top 3 accuracy rate of nearly 100%. For more information, please follow other related articles on the PHP Chinese website!