Important breakthrough! The Westlake University team and Zhejiang Second Hospital jointly realized Chinese decoding of brain-computer interface

Professor Mohammed Sawan’s team at the Advanced Neural Chip Center, Professor Zhang Yue’s team and Professor Zhu Junming’s team from the Natural Language Processing Laboratory jointly released their latest research results: “A high-performance brain-sentence communication designed for logosyllabic language". This research realizes full-spectrum Chinese decoding of brain-computer interface, which to a certain extent fills the gap in international Chinese decoding brain-computer interface technology.

Brain-computer interface (BCI) is recognized as the main battlefield for the cross-integration of life sciences and information technology in the future, and is a research direction with important social value and strategic significance.

Brain-computer interface technology refers to the creation of a connection path for information exchange between the human or animal brain and external devices. Its essence is a new type of information transmission channel that allows information to bypass the original muscle and peripheral nerve pathways. Realize connection with the external world, thereby replacing human movement, language and other functions to a certain extent.

Design and performance of full-spectrum Chinese decoder for brain-computer interface

In August of this year, two back-to-back Nature articles demonstrated the power of brain-computer interfaces in language recovery. However, most of the existing language brain-computer interface technologies are built for alphabetic language systems such as English, and research on language brain-computer interface systems for non-alphabetic systems such as Chinese characters is still blank.

In this study, the research team used stereotactic electroencephalography (SEEG) to collect neural activity signals in the brain corresponding to the pronunciation process of all Mandarin Chinese characters, and combined with deep learning algorithms and language models to achieve full-spectrum pronunciation of Chinese characters. Decoding, establishing a Chinese brain-computer interface system covering the pronunciation of all Chinese Mandarin characters, achieving end-to-end output from brain activity to complete Mandarin sentences.

Chinese, as a pictographic and syllabic language, has more than 50,000 characters, which is significantly different from English, which is composed of 26 letters, so this is a huge challenge for existing language brain-computer interface systems. challenges. In order to solve this problem, over the past three years, the research team has conducted an in-depth analysis of the pronunciation rules and characteristics of the Chinese language itself. Starting from the three elements of initial consonants, tones and finals of Chinese pronunciation syllables, and combining the characteristics of the Pinyin input system, a new language brain-computer interface system suitable for Chinese was designed. The research team built a Chinese speech-SEEG database of more than 100 hours by designing a speech database covering all 407 Chinese Pinyin syllables and Chinese pronunciation characteristics and simultaneously collecting EEG signals. Through artificial intelligence model training, the system built a prediction model for the three elements of Chinese character pronunciation syllables (including initial consonants, tones and finals), and finally integrated all predicted elements through a language model, combining semantic information to generate the most likely Complete Chinese sentences.

The research team evaluated the decoding ability of this brain-computer interface system in a simulated daily Chinese environment. After more than 100 randomly selected complex communication scene decoding tests of 2 to 15 characters, the median character error rate of all participants averaged only 29%, and some participants obtained completely correct sentences through EEG decoding. reached 30%. The relatively efficient decoding performance is due to the excellent performance of the three independent syllable element decoders and the perfect cooperation of the intelligent language model. In particular, in terms of classifying 21 initial consonants, the accuracy of the initial consonant decoder exceeds 40% (more than 3 times the baseline), and the Top 3 accuracy rate reaches almost 100%; while the tone decoder used to distinguish 4 tones The accuracy rate also reached 50% (more than 2 times the baseline). In addition to the outstanding contributions of the three independent syllable element decoders, the powerful automatic error correction and context connection capabilities of the intelligent language model also make the performance of the entire language brain-computer interface system even more outstanding.

This research provides a new perspective for the study of BCI decoding of Chinese, a phonetic language, and also proves that the performance of language brain-computer interface systems can be significantly improved through powerful language models, laying the foundation for future phonetic language neural prostheses. Research provides new directions. This work also indicates that patients with neurological diseases will soon be able to control computer-generated Chinese sentences through their thoughts and regain the ability to communicate!

Reference content

https://www.biorxiv.org/content/10.1101/2023.11.05.562313v1.full.pdf

The above is the detailed content of Important breakthrough! The Westlake University team and Zhejiang Second Hospital jointly realized Chinese decoding of brain-computer interface. For more information, please follow other related articles on the PHP Chinese website!