Home > Article > Technology peripherals > OpenAI uses GPT-4 to explain GPT-2’s 300,000 neurons: This is what wisdom looks like
Although ChatGPT seems to bring humans closer to recreating intelligence, so far we have never fully understood what intelligence is, whether natural or artificial.
It is obviously necessary to understand the principles of intelligence. How to understand the intelligence of large language models? The solution given by OpenAI is: ask what GPT-4 says.
On May 9, OpenAI released its latest research, which used GPT-4 to automatically interpret neuron behavior in large language models and obtained many interesting results.
A simple way to study interpretability is to first understand the various components of the AI model (neurons and attention heads) )doing what. Traditional methods require humans to manually inspect neurons to determine which features of the data they represent. This process is difficult to scale, and applying it to neural networks with hundreds or hundreds of billions of parameters is prohibitively expensive.
So OpenAI proposed an automated method - using GPT-4 to generate and score natural language explanations of neuron behavior and apply it to another language Neurons in the model - Here they chose GPT-2 as the experimental sample and published a data set of interpretations and scores of these GPT-2 neurons.
This technology allows people to use GPT-4 to define and automatically measure the quantitative concept of explainability of AI models: it is used to measure language models using natural language compression and reconstruction The ability of neurons to activate. Due to their quantitative nature, we can now measure progress in understanding the computational goals of neural networks.
OpenAI said that using the benchmark they established, using AI to explain AI can achieve scores close to human levels.
## OpenAI co-founder Greg Brockman also said that we have taken an important step towards using AI to automate alignment research.
Specific methodThe method of using AI to explain AI involves running three steps on each neuron:
Step 1: Use GPT-4 to generate explanations
##Given a GPT-2 neuron , by showing GPT-4 relevant text sequences and activations to generate explanations of its behavior.Explanations of model generation: References to movies, characters, and entertainment.
Step 2: Use GPT-4 to simulate
Use GPT-4 again to simulate the interpreted neural What will Yuan do.
Step 3: Comparison Explanations are scored based on how well simulated activations match real activations - in this case, GPT-4 scored 0.34.
Using its own scoring method, OpenAI began measuring the effectiveness of their technology on different parts of the network and trying to improve the technology for parts that are currently unclear. For example, their technique does not work well with larger models, possibly because later layers are more difficult to interpret.
OpenAI says that while the vast majority of their explanations didn’t score highly, they believe they can now use ML technology to further enhance their ability to generate explanations. For example, they found that the following helped improve their scores:
OpenAI says it is making open source the dataset and visualization tools written by GPT-4 that interpret all 307,200 neurons in GPT-2. At the same time, they also provide code for interpretation and scoring using models publicly available on the OpenAI API. They hope the research community will develop new techniques to generate higher-scoring explanations, as well as better tools to explore GPT-2 through explanations.
They found that more than 1,000 neurons had an explanation score of at least 0.8, meaning they accounted for most of the neuron's top activation behavior according to GPT-4. Most of these well-explained neurons are not very interesting. However, they also found many interesting neurons that GPT-4 did not understand. OpenAI hopes that as explanations improve, they may quickly uncover interesting qualitative insights into model computations.
Here are some examples of neurons being activated in different layers, with higher layers being more abstract:
#It seems that the concepts that GPT understands are different from humans?Future work of OpenAI
Currently, this method still has some limitations, and OpenAI hopes to solve these problems in future work:Ultimately, OpenAI hopes to use models to form, test, and iterate completely general hypotheses, just as explainability researchers do. Additionally, OpenAI hopes to interpret its largest models as a way to detect alignment and security issues before and after deployment. However, there is still a long way to go before that happens.
The above is the detailed content of OpenAI uses GPT-4 to explain GPT-2’s 300,000 neurons: This is what wisdom looks like. For more information, please follow other related articles on the PHP Chinese website!