Your next steps

Choose your focus

Create your 1-pager

Next steps: Apply to roles

Next steps: Technical fellowships

Next steps: Policy fellowships

Next steps: Other fellowships

Next steps: Programs

How does AI think?

Interpretability in practice

Making AI go well

What might success look like?

Building AI safely is hard

What future do you want?

Assuming harm

Building defences

Break the kill chain

Evaluations: AI can, but will it?

How do AI companies test for safety?

Option 1: Anthropic

Option 2: OpenAI

Option 3: Google DeepMind

Option 4: Meta

Can we train AI to be safe?

Feeding AI ‘good’ data

Teaching AI right from wrong

More safety techniques

<Embed url="https://www.youtube.com/embed/EtVsbBd3Rv8"/>

We’ve tried to train safe models, but how do we know if we’ve succeeded?

There are two broad things we want to be able to detect through evaluations:

- **Capabilities**: can a model do X? It finds the upper bounds of what an AI can do, typically by giving the model tasks and seeing how reliably it solves them. E.g. [MMLU](https://www.datacamp.com/blog/what-is-mmlu), [ARC-AGI](https://arcprize.org/blog/launch).
- **Propensities**: will a model do X? It tells us what the model’s behavioural tendencies are, typically by placing the model in different scenarios and seeing which behaviours it tends to exhibit more. E.g. [TruthfulQA](https://truthfulai.org/papers/truthfulqa/), [scheming evals](https://www.antischeming.ai/)

AI companies typically evaluate models:

1. **During training** where you monitor emerging behaviours and capabilities as they develop. This is where interventions are cheapest. You can adjust training data, modify reward signals, or even halt training entirely.
2. **Before deployment** when red teams try to break models, when dangerous capability thresholds are checked, and when go/no-go decisions get made.
3. **Post-deployment** where usage is monitored and suspicious behaviour is flagged.


Pick ONE of the following dangerous capabilities:
- Scheming
- Manipulation
- Cyberattack uplift
- Biorisk uplift

Then, you can use the optional resources below to, in your own words and simple English (no jargon): 

- **Explain step-by-step.** What are different ways we evaluate for this?
- **Describe a technical failure mode**. How robust is this evaluation?
- **Brainstorm fixes.** What technical patches could help? What's fundamentally unfixable?

_Though we have provided multiple options for the resources, we recommend spending 45 minutes reading and 15 minutes writing._

If you're completing this course with a group, post in Slack which capability you selected.


Evaluating for dangerous capabilities

Every AI company claims their models are safe. But what does "safe" actually mean to them? What do they test for? What don't they test for? And how rigorous are these evaluations?

In this section, you'll examine how AI companies actually evaluate their models by looking at their:

- **Risk management frameworks**: Commitments about when to pause or implement additional safety measures
- **System cards**: Documentation of actual evaluations performed on specific models
- **Red team reports**: Results from adversarial testing

You'll see that companies take dramatically different approaches. 

Some test extensively for bioweapon risks while others barely mention them. Some use external evaluators while others rely on internal teams. Understanding these differences is crucial for identifying where safety gaps exist.

**Pick ONE company and examine how they evaluate for the dangerous capability you chose in the previous exercise.**


Answer the following questions about your chosen company and dangerous capability:

1. **Limits**: which specific observations about dangerous capabilities would indicate that it is (or strongly might be) unsafe to continue scaling?
2. **Protections**: what aspects of current protective measures are necessary to contain catastrophic risks from this dangerous capability? 
3. **Evaluation**: what are the procedures for promptly catching early warning signs of dangerous capability limits?
4. **Response**: if the dangerous capability goes past the limits and it’s not possible to improve protections quickly, is the AI developer prepared to pause further capability improvements until protective measures are sufficiently improved, and treat any dangerous models with sufficient caution?
5. **Accountability**: how does the AI developer ensure that the commitments are executed as intended; that key stakeholders can verify that this is happening (or notice if it isn’t); that there are opportunities for third-party critique; and that changes to the framework itself don’t happen in a rushed or opaque way?

_We recommend spending 45 minutes reading and 15 minutes writing._


Technical AI Safety

How do AI companies test for safety?

Resources