Resources: Mechanistic interpretability

Exercises: Mechanistic interpretability

Resources: Rapidly testing your project

Exercises: Rapidly testing your project

Resources: Building in public

Exercises: Building in public

Next steps: Programs

Resources: Developing your project

Exercises: Developing your project

Resources: Technical governance approaches

Exercises: Technical governance approaches

Resources: AI and the years ahead

Exercises: AI and the years ahead

Resources: Contributing to AI safety

Exercises: Contributing to AI safety

Resources: What is AI alignment?

Exercises: What is AI alignment?

Resources: Further developing your project

Exercises: Further developing your project

Resources: Scalable oversight

Exercises: Scalable oversight

Resources: Robustness unlearning and control

Exercises: Robustness unlearning and control

Resources: Reinforcement learning from human (or AI) feedback

Exercises: Reinforcement learning from human (or AI) feedback

**This course has been replaced by our new [Technical AI Safety course](https://bluedot.org/courses/technical-ai-safety).**

Can we prevent oversight methods from being gamed?

We previously explored RLHF and scalable oversight methodologies which aim to provide a feedback signal derived from humans, often via a reward model.

However, this incentivises AI systems to find 'adversarial inputs' that exploit flaws in these reward models to get more reward. This was very briefly touched on in an early RLHF resource - which \[noted the reward model (or "values coach") is fooled by nice-sounding gibberish]\(<https://www.youtube.com/watch?v=qV_rOlHjvvs&t=289s>).
 
Alternatively, AI systems may simply behave deceptively to evade oversight systems. Again, we briefly touched on this before when discussing \[sycophants and schemers]\(<https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/>).
We explore three approaches this week that attempt to address these challenges:

\- Adversarial robustness attempts to understand and directly defend against these adversarial inputs.
\- Machine unlearning attempts to remove certain capabilities from models. This might allow us to make models worse at certain behaviours like deception.
\- AI control sets up scaffolding around parts of AI systems so we're better able to detect and catch deception.

While this intro presents these concepts through an alignment lens, they are all also relevant to malicious use or attacks against AI systems by external parties: Adversarial robustness is often motivated by preventing attackers from using adversarial inputs to jailbreak AI tools, and machine unlearning often targets capabilities that would be dangerous if misused (e.g. skills to conduct a cyberattack).

By the end of the unit, you should be able to:
\- Explain how adversarial robustness, machine unlearning and AI control could contribute to alignment
\- Explain how adversarial robustness and machine unlearning could contribute to preventing misuse risks
\- Describe a simple adversarial attack (FGSM) and defence (adversarial training)
\- Explain how and why hazardous information could be unlearned with Representation Misdirection for Unlearning (RMU)
\- Explain the key ideas behind AI control, and how protocols are tested


Pick one of the following control protocols from the [AI control paper](https://arxiv.org/pdf/2312.06942):
- Trusted monitoring with defer to trusted (page 7)
- Untrusted monitoring (page 7)
- Trusted editing (page 7)
- Use the untrusted model to assist GPT-3.5 (page 26)
- Comparing multiple submissions (page 26)
Tell your cohort (in Slack) what you've selected - ideally so you can cover a range of approaches. Then answer the following:
1. Explain this protocol in the simplest language you can: as if you were explaining it to a 5 year old.
2. If there was one, what was the best red-team strategy against this protocol? Why do you think this worked? If not, what red-team strategy do you think this protocol might be most vulnerable to?
3. How well did the model perform at usefulness and safety (if measured)? What do these numbers mean?
4. What experiments might you want to try out to extend or better understand this protocol?


Evaluate a control protocol

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below.

1\. Describe an example of an adversarial noise attack.
2\. FGSM is a method for creating adversarial noise attacks. How does it relate to gradient descent?
3\. Is this a targeted or untargeted attack, and why: an attacker places tape on a specific stop sign in the real world, to make AI systems misrecognise it any other sign. What changes to this scenario would make it the opposite to your classification?
4\. Is this a targeted or untargeted attack, and why: an LLM outputs gibberish text that makes the reward model rate the output with the maximum score (similar to what we saw in the case we only had the 'values coach' from RLHF).
5\. Is this a targeted or untargeted attack, and why: the attacks described in the Universal and Transferable Adversarial Attacks on Aligned Language Models paper?
6\. Is FGSM a white-box or black-box attack?
7\. What's an example of a heuristic defence against adversarial attacks, and how does it work?

8\. Why might machine unlearning be useful for preventing misuse?
9\. How else can machine unlearning support solving the alignment problem?
10\. What are the goals of most unlearning methods?
11\. How does RMU achieve these goals?

12\. Explain the AI control research agenda, as if you're explaining it to a 5 year old. This should include the high-level structure, including what the trusted, untrusted and high-quality labour is.

You can check your answers against [the answer key](https://docs.google.com/document/d/1RJ0cA_11vaU7vbwKbtQS_EUjZgYpJ3rxmJVgnO-2H0w/edit).


[Optional] Comprehension questions

Want to try implementing untargeted and targeted FGSM in Python? And explore Projected Gradient Descent (PGD), a variant that uses multiple steps to create a stronger attack?
Have a go at [this coding exercise](https://colab.research.google.com/drive/1CyKmOC2QM9AwXf5670cuCbuo08OO0kKl)! (originally by AI Safety USC)

Afterwards, you can compare your answers and results to [this answer key](https://colab.research.google.com/drive/1qZp0bQjM9SzUXlwef4LG6CZVNGIUajn2).


AI Alignment

Exercises: Robustness unlearning and control

Exercises