Resources: Adversarial techniques for scalable oversight

Exercises: Adversarial techniques for scalable oversight

Resources: Reward misspecification and instrumental convergence

Exercises: Reward misspecification and instrumental convergence

Resources: Governance (Alignment 23)

Exercises: Governance (Alignment 23)

Resources: Task decomposition for scalable oversight

Exercises: Task decomposition for scalable oversight

Resources: Careers and Projects

Exercises: Careers and Projects

Next steps: Programs

Resources: Interpretability

Exercises: Interpretability

Resources: Agent foundations

Exercises: Agent foundations

Resources: Artificial General Intelligence

Exercises: Artificial General Intelligence

Resources: Introduction to Machine Learning (23)

Exercises: Introduction to Machine Learning (23)

Resources: Goal misgeneralisation

Exercises: Goal misgeneralisation

You can significantly improve the performance of language models on tricky tasks by asking them to break down the problem in the right way.
This week introduces scalable oversight as an approach to preventing reward misspecification, and discusses one scalable oversight proposal: iterated amplification.
Scalable oversight refers to methods that enable humans to oversee AI systems that are solving tasks too complicated for a single human to evaluate. This week begins by justifying the problem of scalable oversight; we then examine iterated amplification as a potential solution to the problem. Iterated amplification is built around task decomposition: the strategy of training agents to perform well on complex tasks by decomposing them into smaller tasks which can be more easily evaluated and then combining those solutions to produce answers to the full task. Iterated amplification involves repeatedly using task decomposition to train increasingly powerful agents.
We'll examine two other alignment techniques that could work at scale next week.

By the end of the unit, you should be able to:
\- Explain the basic concept of scalable oversight.
\- Understand the concept of iterated amplification as a means of achieving scalable oversight.
    \- Understand task decomposition as a component of iterated amplification using two experimental setups in the curriculum as examples.
\- Understand the assumptions that iterated amplification relies on to work to train aligned, powerful agents.
    \- Explain how it helps, the evidence that it works, and its limitations.


AI Alignment (2023)

Exercises: Task decomposition for scalable oversight