OpenAI employs GPT-4 to autonomously generate descriptions for the actions of neurons in large language models and to evaluate those descriptions. They’ve released a dataset containing these (albeit imperfect) explanations and their corresponding scores for each neuron in GPT-2. GPT-5 capabilities: the model will do even more.
As language models become increasingly advanced and widely utilized, OpenAI’s comprehension of their internal mechanisms remains quite limited. For instance, discerning whether they rely on biased heuristics or engage in deception from their outputs can be challenging. Interpretability research seeks to shed more light on these aspects by delving into the model’s inner workings.
One fundamental strategy for interpretability research involves understanding the functions of individual components, such as neurons and attention heads. Traditionally, this has necessitated manual inspection of neurons to identify the features of the data they represent. However, this method is not scalable, especially when dealing with neural networks comprising tens or hundreds of billions of parameters.
OpenAI proposes an automated process using GPT-4 to generate and evaluate natural language explanations of neuron behavior, which is then applied to neurons in another language model.
This effort is a component of the third pillar in approach to alignment research, which is automating the alignment research work itself. An encouraging aspect of this strategy is its scalability with the rate of AI development. As future models grow (GPT-5, GPT-6 … ) in intelligence and utility as assistants, they will yield increasingly insightful explanations.
While developing the feature it also need to consider Limitations.
How it works
Our methodology consists of running 3 steps on every neuron.
- Step 1: Generate explanation using GPT-4
- Step 2: Simulate using GPT-4
- Step 3: Compare
OpenAI is open-sourcing its datasets and visualization tools that contain GPT-4-generated explanations for all 307,200 neurons in GPT-2. In addition, they releasing the code for creating and evaluating explanations using models that are publicly accessible via the OpenAI API. Engineers anticipate that the research community will devise novel techniques to produce higher-scoring explanations and improve tools for investigating GPT-2 using these explanations.
OpenAI’s findings revealed over 1,000 neurons with explanations that scored at least 0.8, indicating that according to GPT-4, they represent the majority of the neuron’s top-activating behavior. Most of these highly-explained neurons aren’t particularly intriguing. However, they did discover many fascinating neurons that GPT-4 couldn’t fully comprehend. Devs hope that as the quality of explanations enhances, they’ll be able to swiftly extract interesting qualitative insights about model computations.
Outlook
OpenAI’s current methodology does come with several constraints, which they hope to address in subsequent research.
- The focus has been on brief natural language explanations, but it’s possible that neurons may exhibit highly complex behavior that’s difficult to concisely describe. For instance, neurons could be intensely polysemantic (representing a multitude of separate concepts), or they might signify singular concepts that humans don’t comprehend or have terminology for.
- OpenAI aspires to ultimately locate and elucidate entire neural circuits demonstrating complex behaviors, with neurons and attention heads collaborating. However, the existing method only expounds on neuron behavior based on the original text input, without providing insight into its downstream effects. A neuron that activates on periods, for example, could be suggesting that the subsequent word should begin with a capital letter, or it might be incrementing a sentence counter.
- The behavior of neurons was explained without attempting to elucidate the mechanisms that induce that behavior. This implies that even high-scoring explanations could perform poorly with out-of-distribution texts, as they’re essentially describing a correlation.
- The overall procedure is significantly compute-intensive.
OpenAI is enthusiastic about potential extensions and generalizations of their approach. The ultimate goal is to utilize models to form, test, and refine fully general hypotheses in the same way an interpretability researcher would.
Ultimately, they aim to interpret their largest models as a means of detecting alignment and safety issues both prior to and following deployment. However, significant progress needs to be made before these techniques can bring to light behaviors such as dishonesty.
Read more articles in our Blog.