Can we prevent oversight methods from being gamed?

We previously explored RLHF and scalable oversight methodologies which aim to provide a feedback signal derived from humans, often via a reward model.

However, this incentivises AI systems to find 'adversarial inputs' that exploit flaws in these reward models to get more reward. This was very briefly touched on in an early RLHF resource - which \[noted the reward model (or "values coach") is fooled by nice-sounding gibberish]\(<https://www.youtube.com/watch?v=qV_rOlHjvvs&t=289s>).
 
Alternatively, AI systems may simply behave deceptively to evade oversight systems. Again, we briefly touched on this before when discussing \[sycophants and schemers]\(<https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/>).
We explore three approaches this week that attempt to address these challenges:

\- Adversarial robustness attempts to understand and directly defend against these adversarial inputs.
\- Machine unlearning attempts to remove certain capabilities from models. This might allow us to make models worse at certain behaviours like deception.
\- AI control sets up scaffolding around parts of AI systems so we're better able to detect and catch deception.

While this intro presents these concepts through an alignment lens, they are all also relevant to malicious use or attacks against AI systems by external parties: Adversarial robustness is often motivated by preventing attackers from using adversarial inputs to jailbreak AI tools, and machine unlearning often targets capabilities that would be dangerous if misused (e.g. skills to conduct a cyberattack).

By the end of the unit, you should be able to:
\- Explain how adversarial robustness, machine unlearning and AI control could contribute to alignment
\- Explain how adversarial robustness and machine unlearning could contribute to preventing misuse risks
\- Describe a simple adversarial attack (FGSM) and defence (adversarial training)
\- Explain how and why hazardous information could be unlearned with Representation Misdirection for Unlearning (RMU)
\- Explain the key ideas behind AI control, and how protocols are tested


Resources: Robustness unlearning and control

Exercises: Robustness unlearning and control

Pick one of the following control protocols from the [AI control paper](https://arxiv.org/pdf/2312.06942):
- Trusted monitoring with defer to trusted (page 7)
- Untrusted monitoring (page 7)
- Trusted editing (page 7)
- Use the untrusted model to assist GPT-3.5 (page 26)
- Comparing multiple submissions (page 26)
Tell your cohort (in Slack) what you've selected - ideally so you can cover a range of approaches. Then answer the following:
1. Explain this protocol in the simplest language you can: as if you were explaining it to a 5 year old.
2. If there was one, what was the best red-team strategy against this protocol? Why do you think this worked? If not, what red-team strategy do you think this protocol might be most vulnerable to?
3. How well did the model perform at usefulness and safety (if measured)? What do these numbers mean?
4. What experiments might you want to try out to extend or better understand this protocol?


Evaluate a control protocol

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below.

1\. Describe an example of an adversarial noise attack.
2\. FGSM is a method for creating adversarial noise attacks. How does it relate to gradient descent?
3\. Is this a targeted or untargeted attack, and why: an attacker places tape on a specific stop sign in the real world, to make AI systems misrecognise it any other sign. What changes to this scenario would make it the opposite to your classification?
4\. Is this a targeted or untargeted attack, and why: an LLM outputs gibberish text that makes the reward model rate the output with the maximum score (similar to what we saw in the case we only had the 'values coach' from RLHF).
5\. Is this a targeted or untargeted attack, and why: the attacks described in the Universal and Transferable Adversarial Attacks on Aligned Language Models paper?
6\. Is FGSM a white-box or black-box attack?
7\. What's an example of a heuristic defence against adversarial attacks, and how does it work?

8\. Why might machine unlearning be useful for preventing misuse?
9\. How else can machine unlearning support solving the alignment problem?
10\. What are the goals of most unlearning methods?
11\. How does RMU achieve these goals?

12\. Explain the AI control research agenda, as if you're explaining it to a 5 year old. This should include the high-level structure, including what the trusted, untrusted and high-quality labour is.

You can check your answers against [the answer key](https://docs.google.com/document/d/1RJ0cA_11vaU7vbwKbtQS_EUjZgYpJ3rxmJVgnO-2H0w/edit).


(Optional) Comprehension questions

Want to try implementing untargeted and targeted FGSM in Python? And explore Projected Gradient Descent (PGD), a variant that uses multiple steps to create a stronger attack?
Have a go at [this coding exercise](https://colab.research.google.com/drive/1CyKmOC2QM9AwXf5670cuCbuo08OO0kKl)! (originally by AI Safety USC)

Afterwards, you can compare your answers and results to [this answer key](https://colab.research.google.com/drive/1qZp0bQjM9SzUXlwef4LG6CZVNGIUajn2).


(Optional) Attack some networks

AI Alignment

Can we prevent oversight methods from being gamed?

Robustness unlearning and control

How and why might we build future AI systems?

AI and the years ahead

What do we need to do to ensure AI systems do what we want, and why is this difficult?

AI Alignment

Resources: Robustness unlearning and control

Resources (1 hr 40 mins)