Resources: Adversarial techniques for scalable oversight

Exercises: Adversarial techniques for scalable oversight

Resources: Reward misspecification and instrumental convergence

Exercises: Reward misspecification and instrumental convergence

Resources: Governance (Alignment 23)

Exercises: Governance (Alignment 23)

Resources: Task decomposition for scalable oversight

Exercises: Task decomposition for scalable oversight

Resources: Careers and Projects

Exercises: Careers and Projects

Next steps: Programs

Resources: Interpretability

Exercises: Interpretability

Resources: Agent foundations

Exercises: Agent foundations

Resources: Artificial General Intelligence

Exercises: Artificial General Intelligence

Resources: Introduction to Machine Learning (23)

Exercises: Introduction to Machine Learning (23)

Resources: Goal misgeneralisation

Exercises: Goal misgeneralisation

Even without knowing which goals an agent will learn, we can predict some properties of its behavior which would be incentivized under many different goals.
Even without reward misspecification, the rewards used during training don’t allow us to control how agents generalize to new situations. This week we cover the scenarios in which agents in new situations generalize to behaving in competent yet undesirable ways because of learning the wrong goals from previous training: the problem of goal misgeneralisation.
The first two readings define and characterize goal misgeneralisation (also known as inner misalignment in the alignment field). Note that goal misgeneralisation and inner misalignment are roughly equivalent concepts, even though they are defined in slightly different ways. Goal misgeneralisation is defined in terms of behaviour in novel situations, while inner misalignment is defined in terms of the representations learned during training.
The following two readings explore specific hypotheses about how agents which have learned the wrong goals will behave: deceptively gaining high reward, and seeking power on a larger scale.
We then finish with two readings on how these behaviours could lead to catastrophe.

By the end of the unit, you should be able to:
\- Understand the concept of internally-represented goals, and evaluate how policies might learn the 'wrong goals'.
    \- Define goal misgeneralisation.
    \- Consider the inner misalignment framework [note, not a definition but is a term that's used in the alignment field.].
\- Explore the extent to which adversarial training mitigates goal misgeneralisation, and evaluate whether it's a complete solution to the goal misgeneralisation problem.
\- Understand the concept of deception and the extent of its relation to goal misgeneralisation.
\- Understand that 'situational awareness' is necessary to cast bad behaviour in terms of 'maximising the reward signal'.
    \- (Without situational awareness, bad behaviour is just bad generalisation. With it, you could interpret the system as trying to manipulate the environment instrumentally towards maximising reward.)


AI Alignment (2023)

Exercises: Goal misgeneralisation