Resources: Mechanistic interpretability

Exercises: Mechanistic interpretability

Resources: Rapidly testing your project

Exercises: Rapidly testing your project

Resources: Building in public

Exercises: Building in public

Next steps: Programs

Resources: Developing your project

Exercises: Developing your project

Resources: Technical governance approaches

Exercises: Technical governance approaches

Resources: AI and the years ahead

Exercises: AI and the years ahead

Resources: Contributing to AI safety

Exercises: Contributing to AI safety

Resources: What is AI alignment?

Exercises: What is AI alignment?

Resources: Further developing your project

Exercises: Further developing your project

Resources: Scalable oversight

Exercises: Scalable oversight

Resources: Robustness unlearning and control

Exercises: Robustness unlearning and control

Resources: Reinforcement learning from human (or AI) feedback

Exercises: Reinforcement learning from human (or AI) feedback

**This course has been replaced by our new [Technical AI Safety course](https://bluedot.org/courses/technical-ai-safety).**

How might we understand what’s going on inside an AI model?

As modern AI systems grow more capable, there is an increasing need to make their internal reasoning and decision making more interpretable and transparent. This week we'll look at how people are analyzing models' learned representations and weights through methods like circuit analysis, and teasing apart behaviours like superposition with dictionary learning. The final resource looks at performing circuit analysis, identifying a circuit for indirect object identification in GPT-2.

By the end of the unit, you should be able to:
\- Understand what mechanistic interpretability is, and how it could help with the AI Alignment problem
\- Understand key mechanistic interpretability concepts, including circuits and superposition.


This is a brief [writing-to-learn exercise](https://bluedot.org/blog/what-is-writing-to-learn/), expected to be no more than half a page. It is intended to be less intensive than previous exercises given the intensity of reading this week.
Pick one of [the three claims](https://distill.pub/2020/circuits/zoom-in/#three-speculative-claims:~:text=Three%20Speculative%20Claims%20about%20Neural%20Networks) (features, circuits and universality) given in the circuits paper. Then:
1. Summarise what the claim is in your own words. Explain if you think you might be uncertain what the claim is.
2. What evidence or arguments support and reject the claim? (this can be from the circuits paper itself, the other resources, or other information you find)


Do the speculative circuits claims hold up?

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below.

Zoom In: An Introduction to Circuits
1．What are the three key claims about neural networks that the paper introduces and evaluates?
2．How do features and circuits relate?
3．What does it mean to say a feature is a "direction" in a neural network?
4．What are some examples of circuits or features identified in the paper?
5．What evidence is there that image models detect features like curves?
6．What is a polysemantic neuron?
7．Why do polysemantic neurons make interpreting models more difficult?
8．Suggest a possible circuit that might be present in a large language model.

Let's Try To Understand AI Monosemanticity
9．What is "superposition" in neural networks?
10．What does the "sparsity" of features mean?
11．Why is high sparsity likely in practice with large language models?
12．More superposition tends to occur if features are \_\_\_\_\_ [less/more] sparse.
13．Why might disentangling superposition help us interpret neural networks?
14．What type of neural network or algorithm did Anthropic show helped perform dictionary learning to identify learned features in a model?
15．Explain the general approach taken in the 'towards Monosemanticity' paper.
16．Explain what [feature #442](https://transformer-circuits.pub/2023/monosemantic-features/vis/a1.html#feature-442) is in A/1, and suggest a way we could use it.

Interpretability in the Wild (from the further resources)
17．What specific capability in GPT-2 small does this paper analyze?
18．Describe the algorithm the paper proposes GPT-2 is carrying out to perform this task.
19．What are the three major classes of token heads the paper claims implement this algorithm?
20．What is path patching? How did this help the authors identify the relevant heads?

You can check your answers against [the answer key](https://docs.google.com/document/d/1hEfb-gvvyfyPtBLwqy8v2BBUCCBPACc4a3rxhD29A8k/edit).


[Optional] Comprehension Questions

For the more technically inclined, the ARENA curriculum offers [a section on transformer interpretability](https://arena3-chapter1-transformer-interp.streamlit.app/). You may need to read the earlier ARENA content, particularly [chapter 0](https://arena3-chapter0-fundamentals.streamlit.app/), to set up an environment and understand more of the code.
You can get started by [replicating the indirect object identification circuit paper](https://arena3-chapter1-transformer-interp.streamlit.app/[1.3]_Indirect_Object_Identification). This is a difficult exercise, and one that we expect to take at least a day even for experienced ML engineers.
You may be able to find collaborators to work through this together in your cohort, the [#find-collaborators](https://aisafetyfundamentals.slack.com/archives/C0280PH510U) or [#discussion](https://aisafetyfundamentals.slack.com/archives/C025HB1LUE7) Slack channels.
