Resources: Using AI to align AI

Exercises: Using AI to align AI

Resources: What makes aligning AI difficult?

Exercises: What makes aligning AI difficult?

Resources: Contributing to AI safety

Exercises: Contributing to AI safety

Resources: Losing control to AI

Exercises: Losing control to AI

Resources: Understanding how AI systems "think"

Exercises: Understanding how AI systems "think"

In this unit, we will explore what ‘alignment’ means and where it sits in the wider AI safety landscape. We’ll then discuss common arguments about AI safety, as well as concepts such as outer/inner alignment, agents, and convergent instrumental goals.

By the end of the unit, you should be able to:
\- Define AI alignment
\- Explain why maintaining control of AI systems is challenging


In this exercise, you'll explore a random case study and analyse it with the frameworks you've learnt about from the resources. In the session, you'll have to teach one of your fellow participants about your case study so make sure you know it well!

To start, pick one of the following case studies:
- [Racial bias in health algorithms](https://www.ftc.gov/system/files/documents/public_events/1548288/privacycon-2020-ziad_obermeyer.pdf)
- [Keys and Chests environment (section 3.3)](https://ar5iv.labs.arxiv.org/html/2105.14111#page=5:~:text=3.3,Keys%20and%20Chests)
- [The evolved radio (section IV. B and C)](https://people.duke.edu/~ng46/topics/evolved-radio.pdf)
- [Pneumonia chest radiographs ("Discussion" section)](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002683#sec026)
- [OpenAI emergent tool use ("Surprising behaviors" section)](https://openai.com/research/emergent-tool-use#surprisingbehaviors) ([video version](https://www.youtube.com/watch?v=Lu56xVlZ40M))
- [CycleGAN steganography ("Hidden Information" section)](https://ar5iv.labs.arxiv.org/html/1712.02950#:~:text=these%20image%20domains.-,2,Hidden%20Information,-We%20begin%20with)
- [Skin lesion classification ("Nonstandardized and standardized input in AI classification" section)](https://www.sciencedirect.com/science/article/pii/S0022202X18322930)
- [Radiographic COVID-19 detection](https://www.nature.com/articles/s42256-021-00338-7)
- [Atari games (section 5)](https://ar5iv.labs.arxiv.org/html/1802.08842#page=4:~:text=initial%20environment%20conditions.-,5,Qualitative%20analysis,-Visual%20inspection%20and)

Tell your cohort (in Slack) what case study you've selected - ideally so you can cover a range of case studies. Then answer the following:

1. What do you think the human-level intended outcome for the system was?
2. What was the goal specified to the model?
3. What did the model actually do?
4. Is this outer or inner misalignment?
5. Only if outer: What flaw in the reward signal is being exploited? Why does this achieve the goal specified, but not the human-level outcome?
6. Only if inner: What change in distribution occurred between training and deployment? (anthropomorphising the policy here) What learned goal does the model appear to pursue when deployed?
7. Does this failure have any traits that are similar to a scheming or sycophantic model? (the answer will be no for some case studies!)



Misalignment case study

To tackle a problem, it's important to understand it well. This [writing-to-learn exercise](https://bluedot.org/blog/what-is-writing-to-learn/) aims to help reflect on your uncertainties around what the alignment problem is, as well as develop more comprehensive threat models for misalignment scenarios. In the session, you'll discuss your answers to these prompts and other statements about AI safety with your cohort.

Spend ~30 minutes answering the question: **What are my key uncertainties about AI safety and the alignment problem?**

You might want to consider:
- Are there any 'obvious' potential solutions to AI safety that come to mind? (We'd encourage brainstorming a long list first, before trying to analyse them)
- How likely do you think it is that human-level machine intelligence arises in the next 10 years? What about 100 years?
- Do you think AI systems that we build will actually pursue convergent instrumental goals (mentioned at [13:16 in Intro to AI safety](https://www.youtube.com/watch?v=pYXy-A4siMw&t=796s))? Why and why not?
- Will AI systems be safe by default? Why and why not?
- Will we have one big AI system? Many copies of the same powerful AI? Lots of different AI systems?
- Who's intentions or which "values" should we be aligning AI systems with? How would you handle different stakeholders wanting to align AI systems with different intentions or values?

In particular, if you're convinced one way on a particular topic, we'd encourage really digging into the opposite argument. You're also encouraged to browse the optional resources (below), or do your own research to help fill in key gaps in your understanding.


Identifying uncertainties

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below. 

1\. Why is solving the alignment problem insufficient for making AI go well?
2\. For each of the following research areas, identify an [AI risk](https://aisafetyfundamentals.com/blog/ai-risks/) that it seems closely related to and explain why: alignment, moral philosophy, competence, governance, resilience, security.
3\. What is general intelligence?
4\. Summarize Karnofsky's story for how we could stumble into an AI catastrophe.
5\. What about this story feels realistic and unrealistic to you? Be specific!
6\. Explain what sycophantic behaviour is in your own words. Why might this be a problem?
7\. What's the difference between a sycophantic and scheming model?
8\. Why might AI systems try to secure resources, or prevent themselves from being turned off?
9\. Give an example of an AI malfunction from the readings.
10\. How might AI create incentives for war?

You can check your answers against [the answer key](https://docs.google.com/document/d/1by8XAaXySdx-rwo7NkudQRd7HauLnAIhTwq1wj_u99w/edit).


(Optional) Comprehension questions

AI Alignment Fast-Track

Why might we lose control to AI, and why is this a difficult problem to solve?

Losing control to AI

Learn why telling AI systems to do what we want is a surprisingly challenging task.