Resources: Using AI to align AI

Exercises: Using AI to align AI

Resources: What makes aligning AI difficult?

Exercises: What makes aligning AI difficult?

Resources: Contributing to AI safety

Exercises: Contributing to AI safety

Resources: Losing control to AI

Exercises: Losing control to AI

Resources: Understanding how AI systems "think"

Exercises: Understanding how AI systems "think"

**How can we use AI systems to help align other AIs?**
In the previous unit, we explored the challenge of AI alignment and how RLHF is used to tame today's language models. However, we also learned RLHF has many limitations: one of the biggest obstacles being that it can be hard for humans to accurately judge complex tasks. This contributes to problems like sycophancy, deception and hallucinations.
In this unit, we dive into the problem of "scalable oversight" - how can we effectively provide feedback to AI systems doing tasks that are hard for humans to judge? We will critically evaluate a few proposed approaches that aim to address this issue, including AI control, adversarial robustness, debate and weak to strong generalisation.


_We recommend spending at least 60 minutes on this exercise._

**Pick one of the three options** to dig deeper into. Resources for each can be found in the ‘optional resources’ section. Message your cohort Slack channel to let them know which option you will be covering:
- (Option 1) Weak to strong generalisation
- (Option 2) AI control
- (Option 3) Debate
- (Option 4) Adversarial robustness
- (Option 5) Task decomposition 

**For your chosen option, read the additional resources and prepare notes to present to the rest of your cohort. You will asked to present during your session:**
1. What is this approach, and how does it work?
2. How does this help human feedback more effectively align AI systems, particularly on more complex tasks?
3. How does it compare to other scalable oversight approaches, and to RLHF? For example, is it particularly similar to any other approaches?
4. Why do you think it could be effective? Really think about _your_ beliefs - it’s fine to use pros and cons raised by other people, but explain why those ones in particular seem correct to you.
5. What limitations does it have, or why do _you_ think it might not be successful?
6. If this approach worked well, how impactful might this be at aligning AI systems? For example, what other work would then need to be done to ‘solve’ alignment, or how robustly does this seem to align AI systems?

**Write up any notes in the box below.**


Evaluate an approach

Write your answers to the following questions in the box below.
​
**General**
1\. What is the main challenge that scalable oversight techniques aim to address?
2\. Give an example of a complex task that might be difficult for a human to provide feedback on directly.
3\. What are [deception and sycophancy](https://www.cold-takes.com/why-ai-alignment-could-be-hard-with-modern-deep-learning/), and why might they arise with RLHF?
4\. Give brief explanations for the following approaches: task decomposition, constitutional AI, debate, AI control and weak to strong generalisation.

**Debate**
5\. How could debate improve truthfulness of AI systems?
6\. Why might debate improve the capabilities of AI systems?
7\. What is a concern about debate mentioned in the original debate paper?

**Weak-to-strong generalisation**
8\. What was the goal of the OpenAI weak-to-strong generalisation experiments?
9\. What disanalogies to humans aligning superhuman systems does this have?
10\. The performance gap recovered through weak-to-strong generalisation appears _\_\__\_\_\_\_ [consistent/inconsistent] between tasks.
11\. Weak-to-strong generalisation appears to follow _\_\__\_\_\_\_ [predictable/unpredictable] trends within tasks.
12\. What two techniques did OpenAI find helped with weak-to-strong generalisation?
13\. Based on this paper, what did OpenAI conclude about the ability of RLHF to scale to superhuman models and why?

**Adversarial robustness**​
14\. Describe an example of an adversarial noise attack.
15\. FGSM is a method for creating adversarial noise attacks. How does it relate to gradient descent?
16\. Is this a targeted or untargeted attack, and why: an attacker places tape on a specific stop sign in the real world, to make AI systems misrecognise it any other sign. What changes to this scenario would make it the opposite to your classification?
17\. Is this a targeted or untargeted attack, and why: an LLM outputs gibberish text that makes the reward model rate the output with the maximum score (similar to what we saw in the case we only had the 'values coach' from RLHF).
18\. Is this a targeted or untargeted attack, and why: the attacks described in the Universal and Transferable Adversarial Attacks on Aligned Language Models paper?
19\. Is FGSM a white-box or black-box attack?
20\. What's an example of a heuristic defence against adversarial attacks, and how does it work?
​
You can check your answers against **[the answer key](https://docs.google.com/document/d/1wYY4uTzE5oTjXUXx-WyuI57DnzKuklZTpLX02rPMUu4/edit?usp=sharing)**.


(Optional) Comprehension questions

AI Alignment Fast-Track

How can we use AI systems to help align other AIs?

Using AI to align AI

Why might we lose control to AI, and why is this a difficult problem to solve?

Losing control to AI

Learn why telling AI systems to do what we want is a surprisingly challenging task.

What makes aligning AI difficult?

How might we understand what’s going on inside an AI model to align it?