Exercises: Decomposing Tasks for Better Supervision

Resources: Decomposing Tasks for Better Supervision

Exercises: Interpretability 9

Resources: Interpretability 9

Exercises: Track 1: Eliciting Latent Knowledge

Resources: Track 1: Eliciting Latent Knowledge

Exercises: Literature Review or Project Proposal

Resources: Literature Review or Project Proposal

Next steps: Programs

Exercises: Track 3: Science and Deep Learning

Resources: Track 3: Science and Deep Learning

Exercises: Reasoning about Reasoning

Resources: Reasoning about Reasoning

Exercises: Further Understanding the Alignment Problem

Resources: Further Understanding the Alignment Problem

Exercises: Preventing Misgeneralization

Resources: Preventing Misgeneralization

Exercises: Track 2: Agent Foundations

Resources: Track 2: Agent Foundations

Train a two layer MLP to do addition mod 97 (input format is a 2\*97 dimensional vector of each input one-hot encoded and concatenated together, output is a 97 dimensional vector of logits). Give it 30% of the data as training, and use weight decay of 1. Pick a neuron, and plot it as a heatmap over the two inputs, and see if it looks periodic.
- **_Bonus:_** play around with the hyper-parameters and see how dramatically you can get it to grok.
- **_Bonus:_** do this for a transformer, replicate Nanda (2022)’s analysis, then apply it to modular subtraction.


Train a two layer MLP

Replicate the experiment in the Demonstrating Superposition section of Elhage et al. (2022): training a model such that ReLU(W^TWx) is approximately x. Show that as you vary sparsity, the model goes from having one feature per dimension to using superposition.
- **_Bonus:_** Pick your favorite other experiment from the paper and try replicating that (e.g. can you get your network to form a tetrahedron?)


Replicate the experiment

**Problem:** Misaligned models may reason about how to perform in desirable ways during adversarial training (e.g. because we can’t generate compelling unrestricted adversarial inputs). 
**Proposed solution:** Leverage interpretability techniques to detect and penalize undesirable reasoning. (One useful framing: how could interpretability tools be used to extend Debate so that debaters can make arguments about each others’ thought processes?)
### Assumed background
1. [Zoom In: an introduction to circuits (Olah et al., 2020)](https://distill.pub/2020/circuits/zoom-in/) (35 mins)


Alignment 201

Exercises: Interpretability 9

Exercises