Google takes action to rectify the "amnesia" of large models! The feedback attention mechanism helps you "update" the context, and the era of unlimited memory for large models is coming.

Editor | Yi Feng

produced | 51CTO technology stack (WeChat ID: blog51cto)

Google finally took action! We will no longer suffer from the "amnesia" of large models.

TransformerFAM was born, promising to make large models have unlimited memory!

Without further ado, let’s take a look at the “efficacy” of TransformerFAM:

Picture

The large model is processing long context tasks Performance has been significantly improved!

In the above figure, tasks such as Isabelle and NarrativeQA require the model to understand and process a large amount of contextual information and give accurate answers or summaries to specific questions. In all tasks, the model configured with FAM outperforms all other BSWA configurations, and it can be seen that beyond a certain point, the increase in the number of BSWA memory segments cannot continue to improve its memory capabilities.

It seems that on the way to long texts and long conversations, the "unforgettable" of FAM, a big model, does have something to it.

Google researchers introduced FAM, a novel Transformer architecture - Feedback Attention Memory. It uses feedback loops to enable the network to pay attention to its own drift performance, promote the emergence of the Transformer's internal working memory, and enable it to handle infinitely long sequences.

To put it simply, this strategy is a bit like our strategy to artificially combat the "amnesia" of large models: enter the prompt again before each conversation with the large model. It's just that FAM's approach is more advanced. When the model processes a new data block, it will use the previously processed information (that is, FAM) as a dynamically updated context and integrate it into the current processing process again.

In this way, you can well deal with the problem of "forgetting things". Even better, despite the introduction of feedback mechanisms to maintain long-term working memory, FAM is designed to maintain compatibility with pre-trained models without requiring additional weights. So in theory, the powerful memory of the large model does not make it dull or consume more computing resources.

So, how was such a wonderful TransformerFAM discovered? What are the related technologies?

1. From the perspective of challenges, why can TransformerFAM help large models “remember more”?

The concept of Sliding Window Attention (SWA) is crucial to the design of TransformerFAM.

In the traditional Transformer model, the complexity of self-attention (Self-Attention) increases quadratically as the length of the sequence increases, which limits the model's ability to handle long sequences.

"In the movie Memento (2000), the main character suffers from anterograde amnesia, which means he cannot remember what happened in the past 10 minutes, but his long-term memory is intact , he had to tattoo important information on his body to remember them, similar to the current state of large language models (LLMs)," the paper reads.

Screenshots from the movie "Memory", the pictures come from the Internet

Sliding Window Attention (Sliding Window Attention), it is an improved attention Mechanism for processing long sequence data. It is inspired by the sliding window technique in computer science. When dealing with natural language processing (NLP) tasks, SWA allows the model to focus on only a fixed-size window of the input sequence at each time step, rather than the entire sequence. Therefore, the advantage of SWA is that it can significantly reduce the computational effort.

Picture

However, SWA has limitations because its attention span is limited to the window size, which results in the model being unable to consider outside the window. Important information.

TransformerFAM achieves integrated attention, block-level updates, information compression, and global context storage by adding feedback activation to re-input context representation into each block of sliding window attention.

In TransformerFAM, improvements are achieved through feedback loops. Specifically, when processing the current sequence block, the model not only focuses on elements within the current window, but also reintroduces previously processed contextual information (i.e., previous "feedback activation") as additional input into the attention mechanism. In this way, even if the model's attention window slides over the sequence, it is able to maintain memory and understanding of previous information.

So, after these improvements, TransformerFAM gives LLMs the potential to handle infinite length sequences!

2. 作業記憶の大規模モデルを使用して、AGI への移行を継続します

TransformerFAM は研究で前向きな見通しを示しており、これにより AI の長いテキスト タスクの理解と生成の能力が向上することは間違いありません。処理などのパフォーマンスが向上します。文書の要約、ストーリーの作成、Q&A など。

写真

同時に、それがインテリジェントなアシスタントであれ、感情的なパートナーであれ、無制限のメモリを持つ AI はより魅力的に聞こえます。

興味深いことに、TransformerFAM の設計は生物学の記憶メカニズムに触発されており、AGI が追求する自然知能シミュレーションと一致しています。この論文は、神経科学の概念である注意ベースの作業記憶を深層学習の分野に統合する試みです。

TransformerFAM は、フィードバック ループを通じて大規模なモデルに作業記憶を導入し、モデルが短期的な情報を記憶するだけでなく、長期シーケンスにおける重要な情報の記憶を維持できるようにします。

研究者は、大胆な想像力を通じて、現実世界と抽象概念の間に仮説的な橋を架けます。 TransformerFAM のような革新的な成果が生まれ続けるにつれて、技術的なボトルネックは何度も突破され、よりインテリジェントで相互接続された未来がゆっくりと私たちに向かって展開されています。

AIGC の詳細については、次のサイトをご覧ください:

51CTO AI.x コミュニティ

https://www.51cto.com/aigc/

The above is the detailed content of Google takes action to rectify the "amnesia" of large models! The feedback attention mechanism helps you "update" the context, and the era of unlimited memory for large models is coming.. For more information, please follow other related articles on the PHP Chinese website!