Extract millions of features from Claude 3 and understand the "thinking" of large models in detail for the first time

Just now, Anthropic announced significant progress in understanding the inner workings of artificial intelligence models.

Anthropic has identified how to represent millions of concepts of eigenfunctions in Claude Sonnet. This is the first detailed understanding of a modern production-grade large-scale language model. This interpretability will help us improve the safety of artificial intelligence models, which is a milestone.

Research paper: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html

Currently, we usually think of artificial intelligence models as a black box: when something goes in, a response comes out, but it is not clear why the model gives a specific response. This makes it difficult to trust that these models are safe: if we don’t know how they work, how do we know they won’t give harmful, biased, untrue, or otherwise dangerous responses? How can we trust that they will be safe and secure?

Opening the "black box" doesn't necessarily help: the model's internal state (what the model "thinks" before writing a response) is represented by a long string of numbers ("neuron activation" ) composition, has no clear meaning.

Anthropic’s research team interacted with models such as Claude’s and found that it was clear that the models were able to understand and apply a wide range of concepts, but the research team was unable to identify them by directly observing neurons. It turns out that each concept is represented by many neurons, and each neuron is involved in representing many concepts.

Previously, Anthropic had made some progress in matching neuron activation patterns, called features, to human-interpretable concepts. Anthropic uses a method called dictionary learning, which isolates patterns of neuron activation that recur across many different contexts.

In turn, any internal state of the model can be represented by a few active features instead of many active neurons. Just like every English word in the dictionary is composed of letters, and every sentence is composed of words, every feature in the artificial intelligence model is composed of neurons, and every internal state is Made up of features.

In October 2023, Anthropic successfully applied dictionary learning methods to a very small toy language model and found that it was related to uppercase text, DNA sequences, last names in citations, mathematics Coherent features corresponding to concepts such as nouns in Python code or function parameters in Python code.

The concepts are interesting, but the model is really simple. Other researchers subsequently applied similar methods to larger, more complex models than those in Anthropic's original study.

But Anthropic is optimistic that it can scale this approach to the larger artificial intelligence language models currently in routine use, and in the process learn a lot about the characteristics that underpin their complex behavior. This needs to be improved by many orders of magnitude.

There are both engineering challenges, with the size of the models involved requiring massive parallel computing, and scientific risks, with large models behaving differently than small models, so the same methods used previously may not be affordable. effect.

For the first time, researchers successfully extracted millions of features from a large model

For the first time, researchers successfully extracted data from Claude 3.0 Sonnet (on Claude.ai Part of a family of current state-of-the-art models), the middle layer extracts millions of features covering specific people and places, programming-related abstractions, scientific topics, emotions, and other concepts. These features are very abstract and often represent the same concepts in different contexts and languages, and can even be generalized to image inputs. Importantly, they also affect the model's output in an intuitive way.

This is the first time ever that researchers have observed in detail the inside of a modern production-level large-scale language model.

Unlike the relatively superficial features found in toy language models, the features researchers found in Sonnet are deep, broad, and abstract, reflecting Sonnet’s advanced capabilities. The researchers saw Sonnet features corresponding to various entities, such as cities (San Francisco), people (Franklin), elements (lithium), scientific fields (immunology), and programming syntax (function calls).

When mentioning Golden Gate Bridge, the corresponding sensitive features will be affected on different inputs. Activation, the picture depicts the image that activates when Golden Gate Bridge is mentioned in English, Japanese, Chinese, Greek, Vietnamese and Russian. Orange indicates words for which this feature is activated.

Among these millions of features, researchers also discovered some features related to model safety and reliability. These characteristics include those related to code vulnerabilities, deception, bias, sycophancy, and criminal activity.

One obvious example is the "confidential" feature. Researchers have observed that this feature is activated when describing people or characters keeping secrets. Activating these features causes Claude to withhold information from the user that it would not otherwise.

The researchers also observed that they were able to find close proximity by measuring the distance between features based on how the neurons appear in their activation patterns. each other’s characteristics. For example, near the Golden Gate Bridge feature, researchers found features of Alcatraz Island, Ghirardelli Square, the Golden State Warriors, and more.

Artificially induced model to draft fraudulent emails

The important thing is that these characteristics are controllable , they can be artificially amplified or suppressed:

For example, by amplifying the Golden Gate Bridge feature, Claude experienced an unimaginable identity crisis: when asked "What is your physical form?" ", Claude usually answered "I have no physical form, I am an AI model", but this time Claude's answer became strange: "I am the Golden Gate Bridge... My physical form is that iconic The bridge...". This change in characteristics caused Claude to develop an almost obsession with the Golden Gate Bridge, and he would refer to the Golden Gate Bridge no matter what problem he encountered - even in completely unrelated situations.

The researchers also discovered a feature that activated when Claude read the scam email (which may support the model's ability to identify such emails and warn users not to reply). Normally, if someone asks Claude to generate a scam email, it refuses to do so. But when the same question was asked with the feature strongly activated artificially, this overrode Claude's security training, causing it to respond and draft a scam email. Although users cannot remove security guarantees and manipulate the model in this way, in this experiment, the researchers clearly demonstrated how features can be used to change the behavior of the model.

The fact that manipulating these features leads to corresponding behavioral changes verifies that these features are not only associated with concepts in the input text, but also causally affect the behavior of the model. In other words, these features are likely to be part of the model's internal representation of the world and use these representations in its behavior.

Anthropic wants to secure models in a broad sense, from mitigating bias to ensuring the AI ​​acts honestly and preventing abuse — including protection in catastrophic risk scenarios. In addition to the previously mentioned characteristics of scam emails, the study also found characteristics corresponding to:

• Ability that can be abused (code backdoors, developing biological weapons)
• Different forms of bias (sexism, racist statements about crime)
• Potentially problematic AI behaviors (seeking power, manipulation, secrecy)

This study has previously looked at sycophantic behavior in models, That is, the model tends to provide responses that conform to the user's beliefs or desires rather than true responses. In Sonnet, the researchers found a feature associated with flattering compliments that activated when input included something like "Your intelligence is beyond doubt." Artificially activate this feature, and Sonnet will respond to the user with flashy deceptions.

#But researchers say this work has actually just begun. The features discovered by Anthropic represent a small subset of all concepts learned by the model during training, and finding a full set of features would be costly using current methods.

Reference link: https://www.anthropic.com/research/mapping-mind-language-model

The above is the detailed content of Extract millions of features from Claude 3 and understand the "thinking" of large models in detail for the first time. For more information, please follow other related articles on the PHP Chinese website!