Exercises: Decomposing Tasks for Better Supervision

Resources: Decomposing Tasks for Better Supervision

Exercises: Interpretability 9

Resources: Interpretability 9

Exercises: Track 1: Eliciting Latent Knowledge

Resources: Track 1: Eliciting Latent Knowledge

Exercises: Literature Review or Project Proposal

Resources: Literature Review or Project Proposal

Next steps: Programs

Exercises: Track 3: Science and Deep Learning

Resources: Track 3: Science and Deep Learning

Exercises: Reasoning about Reasoning

Resources: Reasoning about Reasoning

Exercises: Further Understanding the Alignment Problem

Resources: Further Understanding the Alignment Problem

Exercises: Preventing Misgeneralization

Resources: Preventing Misgeneralization

Exercises: Track 2: Agent Foundations

Resources: Track 2: Agent Foundations

Use [the tool mentioned by Ziegler et al.](https://rr-data.herokuapp.com/talk-to-filtered-transformer) to find a violent completion which is misclassified by their model.



Misclassified Violent Completion 

**Problem**: we won’t be able to train models to generalize their goals to new situations in desirable ways; instead, they’ll learn to be [deceptively aligned](https://bounded-regret.ghost.io/ml-systems-will-have-weird-failure-modes-2/). 
**Some proposed solutions**: generate novel inputs which we expect to lead to misaligned behavior (known as “unrestricted adversarial examples”), and train models to behave well even on those; or else train models to generalize well directly.
### Assumed background
1. Either of:
    1. [Goal misgeneralization: why correct specifications aren’t enough for correct goals (Shah et al., 2022)](https://arxiv.org/abs/2210.01790) **(only sections 1-4)** (30 mins)
    2. [Goal misgeneralization in deep reinforcement learning (Langosco et al., 2022)](https://arxiv.org/abs/2105.14111) **(ending after section 3.3) **(25 mins)
2. Either of:
    1. [Discovering language model behaviors with model-written evaluations: blog post (Perez et al., 2022)](https://www.alignmentforum.org/posts/yRAo2KEGWenKYZG9K/discovering-language-model-behaviors-with-model-written) (15 mins)
    2. [Red-teaming language models with language models: blog post (Perez et al., 2022)](https://www.deepmind.com/blog/red-teaming-language-models-with-language-models) (10 mins)


Alignment 201

Exercises: Preventing Misgeneralization

Exercises