Resources: Mechanistic interpretability

Exercises: Mechanistic interpretability

Resources: Rapidly testing your project

Exercises: Rapidly testing your project

Resources: Building in public

Exercises: Building in public

Resources: Developing your project

Exercises: Developing your project

Resources: Technical governance approaches

Exercises: Technical governance approaches

Resources: AI and the years ahead

Exercises: AI and the years ahead

Resources: Contributing to AI safety

Exercises: Contributing to AI safety

Resources: What is AI alignment?

Exercises: What is AI alignment?

Resources: Further developing your project

Exercises: Further developing your project

Resources: Scalable oversight

Exercises: Scalable oversight

Resources: Robustness unlearning and control

Exercises: Robustness unlearning and control

Resources: Reinforcement learning from human (or AI) feedback

Exercises: Reinforcement learning from human (or AI) feedback

**This course has been replaced by our new [Technical AI Safety course](https://bluedot.org/courses/technical-ai-safety).**

What do we need to do to ensure AI systems do what we want, and why is this difficult?

In the last unit, we explored the potential impacts of transformative AI. In this unit, we will explore the challenges involved in ensuring those impacts are positive, or we at least avoid the most negative impacts of powerful AI systems.

We explore what ‘alignment’ means, and where it sits in the wider AI safety landscape. We’ll then discuss common arguments about AI safety, as well as concepts such as outer/inner alignment, agents, and convergent instrumental goals.

By the end of the unit, you should be able to:
\- Describe the key areas of AI safety, including alignment, moral philosophy, competence, and governance
\- Understand how some people have classified some types of alignment failures (outer vs inner alignment, sycophants vs schemers)


In this exercise, you'll explore a random case study and analyse it with the frameworks you've learnt about from the resources. In the session, you'll have to teach one of your fellow participants about your case study so make sure you know it well!

To start, pick one of the following case studies:
- [Racial bias in health algorithms](https://www.ftc.gov/system/files/documents/public_events/1548288/privacycon-2020-ziad_obermeyer.pdf)
- [Keys and Chests environment (section 3.3)](https://ar5iv.labs.arxiv.org/html/2105.14111#page=5:~:text=3.3,Keys%20and%20Chests)
- [The evolved radio (section IV. B and C)](https://people.duke.edu/~ng46/topics/evolved-radio.pdf)
- [Pneumonia chest radiographs ("Discussion" section)](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002683#sec026)
- [OpenAI emergent tool use ("Surprising behaviors" section)](https://openai.com/research/emergent-tool-use#surprisingbehaviors) ([video version](https://www.youtube.com/watch?v=Lu56xVlZ40M))
- [CycleGAN steganography ("Hidden Information" section)](https://ar5iv.labs.arxiv.org/html/1712.02950#:~:text=these%20image%20domains.-,2,Hidden%20Information,-We%20begin%20with)
- [Skin lesion classification ("Nonstandardized and standardized input in AI classification" section)](https://www.sciencedirect.com/science/article/pii/S0022202X18322930)
- [Radiographic COVID-19 detection](https://www.nature.com/articles/s42256-021-00338-7)
- [Atari games (section 5)](https://ar5iv.labs.arxiv.org/html/1802.08842#page=4:~:text=initial%20environment%20conditions.-,5,Qualitative%20analysis,-Visual%20inspection%20and)

Tell your cohort (in Slack) what case study you've selected - ideally so you can cover a range of case studies. Then answer the following:

1. What do you think the human-level intended outcome for the system was?
2. What was the goal specified to the model?
3. What did the model actually do?
4. Is this outer or inner misalignment?
5. Only if outer: What flaw in the reward signal is being exploited? Why does this achieve the goal specified, but not the human-level outcome?
6. Only if inner: What change in distribution occurred between training and deployment? (anthropomorphising the policy here) What learned goal does the model appear to pursue when deployed?
7. Does this failure have any traits that are similar to a scheming or sycophantic model? (the answer will be no for some case studies!)


Misalignment case study

To tackle a problem, it's important to understand it well. This [writing-to-learn exercise](https://bluedot.org/blog/what-is-writing-to-learn/) aims to help reflect on your uncertainties around what the alignment problem is, as well as develop more comprehensive threat models for misalignment scenarios. In the session, you'll discuss your answers to these prompts and other statements about AI safety with your cohort.

Spend ~30 minutes answering the question: **What are my key uncertainties about AI safety and the alignment problem?**

You might want to consider:
- Are there any 'obvious' potential solutions to AI safety that come to mind? (We'd encourage brainstorming a long list first, before trying to analyse them)
- How likely do you think it is that human-level machine intelligence arises in the next 10 years? What about 100 years?
- Do you think AI systems that we build will actually pursue convergent instrumental goals (mentioned at [13:16 in Intro to AI safety](https://www.youtube.com/watch?v=pYXy-A4siMw&t=796s))? Why and why not?
- Will AI systems be safe by default? Why and why not?
- Will we have one big AI system? Many copies of the same powerful AI? Lots of different AI systems?
- Who's intentions or which "values" should we be aligning AI systems with? How would you handle different stakeholders wanting to align AI systems with different intentions or values?

In particular, if you're convinced one way on a particular topic, we'd encourage really digging into the opposite argument. You're also encouraged to browse the optional resources (below), or do your own research to help fill in key gaps in your understanding.


Identifying uncertainties

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below. 

1\. Why is solving the alignment problem insufficient for making AI go well?
2\. For each of the following research areas, identify an [AI risk](https://aisafetyfundamentals.com/blog/ai-risks/) that it seems closely related to and explain why: alignment, moral philosophy, competence, governance, resilience, security.
3\. What is general intelligence?
4\. Summarize Karnofsky's story for how we could stumble into an AI catastrophe.
5\. What about this story feels realistic and unrealistic to you? Be specific!
6\. Explain what sycophantic behaviour is in your own words. Why might this be a problem?
7\. What's the difference between a sycophantic and scheming model?
8\. Why might AI systems try to secure resources, or prevent themselves from being turned off?
9\. Give an example of an AI malfunction from the readings.
10\. How might AI create incentives for war?

You can check your answers against [the answer key](https://docs.google.com/document/d/1by8XAaXySdx-rwo7NkudQRd7HauLnAIhTwq1wj_u99w/edit).


AI Alignment

Exercises: What is AI alignment?

Exercises