search
HomeTechnology peripheralsAIBreak the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

For many years, we have been unable to understand how artificial intelligence makes decisions and produces output

Model developers can only decide on algorithms, data, and finally get the model Output results, and the middle part - how the model outputs results based on these algorithms and data, becomes an invisible "black box".

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

So there is a joke like "model training is like alchemy".

But now, the model black box is finally interpretable!

The research team from Anthropic extracted the interpretable features of the most basic unit neurons in the model’s neural network.

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

This will be a landmark step for mankind to uncover the black box of AI.

Anthropic expressed excitedly:

"If we can understand how the neural network in the model works, then we can diagnose the faults of the model. Patterns, design fixes, and safe adoption by businesses and society will become a within-reaching reality!"

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

at Anthropic In the latest research report "Towards Monosemanticity: Using Dictionary Learning to Decompose Language Models", researchers used dictionary learning methods to successfully decompose a layer containing 512 neurons into more than 4,000 interpretable features

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

Research report address: https://transformer-circuits.pub/2023/monosemantic-features/index.html

These features represent DNA sequences, legal language, HTTP requests, Hebrew text, and nutrition instructions, etc.

When we look at the activation of a single neuron in isolation, we It is impossible to see most of these model properties

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

Most neurons are "polysemantic", meaning that a single neuron is There is no consistent correspondence between network behaviors

For example, in a small language model, a single neuron is active in many unrelated contexts, including: academic citations, English Conversations, HTTP requests, and Korean text.

In the classic vision model, a single neuron responds to the face of a cat and the front of a car.

In different contexts, many studies have demonstrated that the activation of a neuron can have different meanings

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

One potential reason is that the polysemantic nature of neurons is due to the additive effect. This is a hypothetical phenomenon whereby neural networks represent independent features of data by assigning each feature its own linear combination of neurons, and the number of such features exceeds the number of neurons

If each feature is regarded as a vector on a neuron, then the feature set forms an overcomplete linear basis for the activation of network neurons.

In Anthropic’s previous Toy Models of Superposition paper, it was proved that sparsity can eliminate ambiguity in neural network training and help the model better understand features. relationship between them, thereby reducing the uncertainty of the source features of the activation vector and making the model’s predictions and decisions more reliable.

This concept is similar to the idea in compressed sensing, where the sparsity of the signal allows the complete signal to be restored from limited observations.

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

But among the three strategies proposed in Toy Models of Superposition:

# (1) Create a model without superposition, Perhaps activation sparsity can be encouraged;

(2) In models that exhibit superposition states, dictionary learning is used to find overcomplete features

(3) relies on a hybrid approach that combines the two.

What needs to be rewritten is: method (1) cannot solve the problem of ambiguity, while method (2) is prone to serious overfitting

So this time Anthropic researchers used a weak dictionary learning algorithm called a sparse autoencoder to generate learned features from the trained model that provide better performance than model neurons itself a more unitary unit of semantic analysis.

Specifically, the researchers adopted an MLP single-layer transformer with 512 neurons, and finally trained a sparse autoencoder on MLP activations from 8 billion data points. Decomposing MLP activations into relatively interpretable features, expansion factors range from 1× (512 features) to 256× (131,072 features).

To verify that the features found in this study are more interpretable than the model's neurons, we conducted a blind review and asked a human evaluator to evaluate their interpretability. Rating

As you can see, the feature (red) has a much higher score than the neuron (cyan).

It has been shown that the features discovered by the researchers are easier to understand relative to the neurons inside the model

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

Additionally, the researchers adopted an "automated interpretability" approach by using a large language model to generate a short description of a small model's features and having another model score that description based on its ability to predict feature activation. .

Likewise, features score higher than neurons, demonstrating a consistent interpretation of the activation of features and their downstream effects on model behavior.

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

Moreover, these extracted features also provide a targeted method to guide the model.

As shown in the figure below, artificially activating features can cause model behavior to change in predictable ways.

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

The following is a visualization of the extracted interpretability features:

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

Click on the feature list on the left to interactively explore the feature space in the neural network

Research Report Summary

This research report from Anthropic, Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, can be divided into four parts.

Problem setting, the researchers introduced the research motivation and elaborated on the trained transfomer and sparse autoencoder.

Detailed investigation of individual features proves that several features found in the study are functionally specific causal units.

Through global analysis, we conclude that the typical features are interpretable and they are able to explain important components of the MLP layer

Phenomenon analysis describes several properties of features, including feature segmentation, universality, and how they form systems similar to "finite state automata" to achieve complex behaviors.

The conclusions include the following 7:

Sparse autoencoder has the ability to extract relatively single semantic features

Sparse autoencoders are able to generate interpretable features that are actually invisible in the basis of neurons

3. Sparse Autoencoders Features can be used to intervene and guide the generation of transformers.

4. Sparse autoencoders can generate relatively general features.

As the size of the autoencoder increases, features tend to "split". After rewriting: As the size of the autoencoder increases, features show a trend of "splitting"

#6. Only 512 neurons can represent thousands of features

7. These features are connected together, similar to a "finite state automaton" system, to achieve complex behaviors, as shown in the figure below

Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier

Specific details can be found in the report.

Anthropic believes that to replicate the success of the small model in this research report to a larger model, the challenge we face in the future will no longer be a scientific problem, but an engineering problem

Achieving interpretability on large models requires more effort and resources in engineering to overcome the challenges posed by model complexity and size

Includes developing new tools, techniques and methods to cope with the challenges of model complexity and data scale; it also includes building scalable interpretive frameworks and tools to adapt to the needs of large-scale models.

This will become the latest trend in interpretive artificial intelligence and large-scale deep learning research

The above is the detailed content of Break the black box of large models and completely decompose neurons! OpenAI rival Anthropic breaks down AI unexplainability barrier. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
解读CRISP-ML(Q):机器学习生命周期流程解读CRISP-ML(Q):机器学习生命周期流程Apr 08, 2023 pm 01:21 PM

译者 | 布加迪审校 | 孙淑娟目前,没有用于构建和管理机器学习(ML)应用程序的标准实践。机器学习项目组织得不好,缺乏可重复性,而且从长远来看容易彻底失败。因此,我们需要一套流程来帮助自己在整个机器学习生命周期中保持质量、可持续性、稳健性和成本管理。图1. 机器学习开发生命周期流程使用质量保证方法开发机器学习应用程序的跨行业标准流程(CRISP-ML(Q))是CRISP-DM的升级版,以确保机器学习产品的质量。CRISP-ML(Q)有六个单独的阶段:1. 业务和数据理解2. 数据准备3. 模型

人工智能的环境成本和承诺人工智能的环境成本和承诺Apr 08, 2023 pm 04:31 PM

人工智能(AI)在流行文化和政治分析中经常以两种极端的形式出现。它要么代表着人类智慧与科技实力相结合的未来主义乌托邦的关键,要么是迈向反乌托邦式机器崛起的第一步。学者、企业家、甚至活动家在应用人工智能应对气候变化时都采用了同样的二元思维。科技行业对人工智能在创建一个新的技术乌托邦中所扮演的角色的单一关注,掩盖了人工智能可能加剧环境退化的方式,通常是直接伤害边缘人群的方式。为了在应对气候变化的过程中充分利用人工智能技术,同时承认其大量消耗能源,引领人工智能潮流的科技公司需要探索人工智能对环境影响的

找不到中文语音预训练模型?中文版 Wav2vec 2.0和HuBERT来了找不到中文语音预训练模型?中文版 Wav2vec 2.0和HuBERT来了Apr 08, 2023 pm 06:21 PM

Wav2vec 2.0 [1],HuBERT [2] 和 WavLM [3] 等语音预训练模型,通过在多达上万小时的无标注语音数据(如 Libri-light )上的自监督学习,显著提升了自动语音识别(Automatic Speech Recognition, ASR),语音合成(Text-to-speech, TTS)和语音转换(Voice Conversation,VC)等语音下游任务的性能。然而这些模型都没有公开的中文版本,不便于应用在中文语音研究场景。 WenetSpeech [4] 是

条形统计图用什么呈现数据条形统计图用什么呈现数据Jan 20, 2021 pm 03:31 PM

条形统计图用“直条”呈现数据。条形统计图是用一个单位长度表示一定的数量,根据数量的多少画成长短不同的直条,然后把这些直条按一定的顺序排列起来;从条形统计图中很容易看出各种数量的多少。条形统计图分为:单式条形统计图和复式条形统计图,前者只表示1个项目的数据,后者可以同时表示多个项目的数据。

自动驾驶车道线检测分类的虚拟-真实域适应方法自动驾驶车道线检测分类的虚拟-真实域适应方法Apr 08, 2023 pm 02:31 PM

arXiv论文“Sim-to-Real Domain Adaptation for Lane Detection and Classification in Autonomous Driving“,2022年5月,加拿大滑铁卢大学的工作。虽然自主驾驶的监督检测和分类框架需要大型标注数据集,但光照真实模拟环境生成的合成数据推动的无监督域适应(UDA,Unsupervised Domain Adaptation)方法则是低成本、耗时更少的解决方案。本文提出对抗性鉴别和生成(adversarial d

数据通信中的信道传输速率单位是bps,它表示什么数据通信中的信道传输速率单位是bps,它表示什么Jan 18, 2021 pm 02:58 PM

数据通信中的信道传输速率单位是bps,它表示“位/秒”或“比特/秒”,即数据传输速率在数值上等于每秒钟传输构成数据代码的二进制比特数,也称“比特率”。比特率表示单位时间内传送比特的数目,用于衡量数字信息的传送速度;根据每帧图像存储时所占的比特数和传输比特率,可以计算数字图像信息传输的速度。

数据分析方法有哪几种数据分析方法有哪几种Dec 15, 2020 am 09:48 AM

数据分析方法有4种,分别是:1、趋势分析,趋势分析一般用于核心指标的长期跟踪;2、象限分析,可依据数据的不同,将各个比较主体划分到四个象限中;3、对比分析,分为横向对比和纵向对比;4、交叉分析,主要作用就是从多个维度细分数据。

聊一聊Python 实现数据的序列化操作聊一聊Python 实现数据的序列化操作Apr 12, 2023 am 09:31 AM

​在日常开发中,对数据进行序列化和反序列化是常见的数据操作,Python提供了两个模块方便开发者实现数据的序列化操作,即 json 模块和 pickle 模块。这两个模块主要区别如下:json 是一个文本序列化格式,而 pickle 是一个二进制序列化格式;json 是我们可以直观阅读的,而 pickle 不可以;json 是可互操作的,在 Python 系统之外广泛使用,而 pickle 则是 Python 专用的;默认情况下,json 只能表示 Python 内置类型的子集,不能表示自定义的

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

Atom editor mac version download

Atom editor mac version download

The most popular open source editor

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

DVWA

DVWA

Damn Vulnerable Web App (DVWA) is a PHP/MySQL web application that is very vulnerable. Its main goals are to be an aid for security professionals to test their skills and tools in a legal environment, to help web developers better understand the process of securing web applications, and to help teachers/students teach/learn in a classroom environment Web application security. The goal of DVWA is to practice some of the most common web vulnerabilities through a simple and straightforward interface, with varying degrees of difficulty. Please note that this software