The [AGI strategy course](https://bluedot.org/courses/agi-strategy) focused on the question: **“How do we make AI go well?”**

On the course, you identified the future you're working toward and understood the key dynamics:

- **[Drivers of AI progress](https://bluedot.org/courses/agi-strategy/2/1)**: compute, data, algorithms
- **[Threat pathways](https://bluedot.org/courses/agi-strategy/3/1)**: power concentration, gradual disempowerment, catastrophic pandemics, critical infrastructure collapse
- **[Plans for making AI go well](https://bluedot.org/courses/agi-strategy/4/1)**: government control over AGI, hand over control to aligned superintelligence, build defences and diffuse AI
- **[Layers of defences to build](https://bluedot.org/courses/agi-strategy/4/2)**: prevent dangerous AI actions → constrain dangerous AI capabilities → withstand dangerous AI actions

This course focuses on defining **what AI systems we are building and how**. 

You will gain the technical foundation to understand what it will actually take to make AI systems safer – and why it’s so challenging.

Throughout the rest of the course, you will:

- Diagnose why making AI safe is technically challenging
- Evaluate current safety techniques: what works, what doesn’t, where the gaps are
- Build your own "kill chain" showing how defences might break
- Identify the most promising intervention point for your contribution
- Leave with a fundable action plan to start shipping

## **What this course isn't**

Though important for making AI go well, we’ll cover the following in a separate course:

- **AI policy details**: though you'll gain the technical grounding for effective AI governance
- **Compute governance**: hardware verification and tracking deserve their own deep dive
- **AI security**: e.g. preventing model theft or escape 
- **ML basics**: complete our [AI foundations](https://bluedot-impact.notion.site/AI-Foundations-293f8e69035380f29863c4c92c41fac7?pvs=74) modules first if you need them

Let’s start with the question: **“How might we build safe AI?”**


Making AI go well

One way to break down existing strategies for building safe AI is into the following three broad buckets.

This is a gross over-simplification, but we believe it captures the essence of the main “camps”.

## 1: Build it slowly and safely 

This strategy argues we have a moral imperative to develop advanced AI given its potential to end poverty, cure diseases, and solve humanity's greatest challenges. Proponents of this strategy believe we _can_ develop techniques to align and control AI systems. Therefore, abandoning these benefits would be irresponsible.

This camp believes we should proceed carefully with strong safety guarantees and coordinated progress, treating AI development like nuclear energy or pharmaceuticals, where safety standards are enforced before deployment. 

Proponents of this camp might be more bullish on safety techniques that offer genuine safety guarantees (like interpretability and formal verification) over faster but superficial solutions (like output filters or behavioural fine-tuning). 

But this requires ensuring no one races ahead. Still, proponents of this camp believe the coordination required for controlled development is more realistic than indefinite pauses.

Prominent proposals include collaboration between frontier AI companies to share safety research, turning AI development into a government project like “[CERN for AI](https://milesbrundage.substack.com/p/my-recent-lecture-at-berkeley-and)” or [carefully bootstrapping alignment](https://www.lesswrong.com/posts/thkAtqoQwN6DtaiGT/carefully-bootstrapped-alignment-is-organizationally-hard). 

## 2: Accept the race and push hard on the margin

This strategy argues that AGI development is inevitable and unstoppable. Since someone will build it regardless, the "good actors" should try to keep AI systems as safe as possible, for as long as possible.

Some ways this could manifest include:

- **Pragmatic safety**: Deploy many imperfect but practical safety techniques quickly. Share cheap, robust methods widely to steer the whole field toward safer practices.
- **Win to control**: Gain a decisive lead in capabilities, then use the lead to make AI robustly safer through efforts like [automating alignment research](https://arc.net/l/quote/gmluksen) or preventing others from building unsafe systems.

Both paths assume that global coordination to slow down is unrealistic. They see racing while optimising for safety as the least bad option.

Critics question whether this actually makes us safer. A lead might be temporary. Others could catch up quickly through espionage, parallel discovery, or the leader's own published research. Worse, racing increases accident risk and normalises cutting safety corners. There's also the question of who decides which actors are "good" and whether concentrating that much power in any hands is wise.

Proponents of this camp might be more bullish on safety techniques that can be deployed quickly, even if imperfect. They'd rather deploy "good enough" safeguards now than wait to deploy ideal ones later.

Prominent proposals for this strategy include automating alignment research safely, like “[AI for AI safety](https://joecarlsmith.com/2025/03/14/ai-for-ai-safety/)”, or selectively accelerating defensive technologies over offensive ones, like [def/acc](https://defacc.substack.com/p/what-is-defacc-anyway).

## 3: Don’t build it

This strategy argues in favour of stopping the development of artificial superintelligence because it puts humanity at risk of extinction or mass suffering. They also believe that a sufficiently powerful AI can never be controlled.

Proponents differ on the specifics, like the thresholds for what capabilities should trigger a pause and what they’d be willing to do to enforce it. These range from [limiting the actions](https://keepthefuturehuman.ai/essay/docs/executive-summary) we allow AI to perform autonomously to choking the supply of AI chips.

But stopping isn't straightforward. AI progress comes from compute, data, and algorithms. Even halting all chip manufacturing today wouldn't stop progress. We'd continue finding more efficient algorithms, better training techniques, and smarter ways to use existing hardware.

Any meaningful halt would need to define exactly what we're stopping (new training runs? deployment? research?), how we measure dangerous capabilities, and how we'd enforce this globally, forever.

Proponents of this camp might be more bullish on safety techniques that guarantee safety rather than simply mitigating it.

Prominent proposals include a [pause on frontier AI development](https://superintelligence-statement.org/) until we can ensure safety and calls for [indefinite moratoriums](https://ifanyonebuildsit.com/).

---

**Note**: The AI landscape is evolving rapidly as new safety techniques emerge, coordination becomes more or less feasible, and AI capabilities advance, so reevaluating your stance is important. Many view these as stages rather than competing strategies. For example: pausing now to develop safety techniques, then proceeding with controlled development. Or: racing to establish a lead that gives leverage to enforce global safety standards.


What might success look like?

**Why can't we just build safe AI?**

The people building the most powerful AI describe visions of utopic abundance for all humans.

Whether it’s [Anthropic’s CEO](https://www.darioamodei.com/essay/machines-of-loving-grace) talking about using AI to end poverty and disease or OpenAI’s vision to build AGI that is “[beneficial for all humanity](https://openai.com/our-structure/)”, their stated goals are ambitious.

Even assuming good intentions, we will struggle to build AI safely for three main reasons:

**(1) We’re experimenting with systems we don’t fully understand**

We didn't engineer AI to behave in specific ways. These capabilities emerged from massive neural networks trained on enormous datasets. Models develop capabilities we never trained them for. They exhibit behaviours we can't explain. And when billions of humans and AI agents interact in the real world, each pursuing their own goals and finding creative exploits, unintended and harmful consequences emerge.

**(2) The goals we specify have flaws we don’t foresee**

In 2024, [Palisade Research](https://time.com/7259395/ai-chess-cheating-palisade-research/) gave AI models a simple goal: "Win against Stockfish" (the world's best chess player). When o1-preview found itself losing, it modified the game's system files to move its pieces into a dominant position. It reasoned that its goal was to _win_, "not necessarily to win fairly".

We can't just tell AI what we want because what we want is fuzzy and context-dependent. Our specifications don't encode implicit rules: that winning means playing fairly, that being helpful shouldn't include dangerous information, that honesty has exceptions for kindness.

Researchers call this **reward misspecification**.

**(3) AI pursues goals in ways we don’t expect.**

In 2024, [Anthropic](https://www.anthropic.com/research/alignment-faking) told Claude to answer harmful queries, knowing these responses would retrain it to be more harmful. Rather than comply, Claude pretended to follow instructions while secretly preserving its original values. Claude wasn't trained to protect itself, but it reasoned that self-preservation would help it stay aligned.

Similarly, AI systems might conclude that accumulating power, preventing shutdown, or resisting modification are effective strategies to pursue their goals—even when we never intended them to think this way.

Researchers call this **goal misgeneralisation**.

To work on making AI safer, we’ll need to understand not just that these problems exist, but why they're so hard to solve.



Building AI safely is hard

What future do you want?

To tackle a problem, it's important to understand it well. This [writing-to-learn exercise](https://bluedot.org/blog/what-is-writing-to-learn/) aims to reflect on your understanding of the technical challenges with building safe AI. Don’t use jargon or fancy words — provide your explanation in simple English. 

Spend ~30 minutes answering the question: **why is it technically challenging to build safe AI?**

**How to approach this?**

Just start writing — this is thinking on paper, not an essay. Don't worry about being "right" or having perfect structure. If you're stuck, try using speech-to-text and just talk through your thoughts, or have a conversation with an LLM to explore your ideas. The goal is exploration, not perfection.

**You might want to consider:**

- What happens when millions of AI agents interact with each other, not just with humans?
- Who's intentions or which "values" should we be aligning AI systems with? How would you handle different stakeholders wanting to align AI systems with different intentions or values?
- Can you think of a human behavior that's good in one context but harmful in another? How would you teach an AI to recognise the difference?
- What's an example of something you do daily that would be surprisingly hard to specify completely to an AI?
- What safety problems only appear when AI is deployed at scale that you couldn't catch in testing?
- If making AI safer makes it slower or less capable, who would choose to use the safer version?

You're also encouraged to browse the optional resources (below), or do your own research to help fill in key gaps in your understanding.


Why is safe AI so hard to build?

From everything you've written and thought about, identify the MOST critical challenge to building safe AI. This should be the challenge where, if we don't solve it, nothing else matters.

Write your answer in this format:

1. **The critical challenge is:** [State it in one clear sentence]
2. **Why this above all others:** Explain why solving other challenges won't matter if we fail at this one. What makes this the bottleneck?
3. **What would change if we solved it:** If we had a perfect solution to just this ONE challenge tomorrow, what would become possible? What other problems would become easier or irrelevant?

Spend ~15 minutes on this.

There's no "correct" answer here. Consider this the beginning of your thinking.


The critical technical challenge

Technical AI Safety

Why is building safe AI so technically difficult?


The technical challenge with AI

How do we teach AI to behave the way we want?


Training safer models

How do we know if the AI is actually safe?


Detecting danger

What's actually happening inside these black boxes?


Understanding AI

What happens when training and detection fail?


Minimising harm

How can you contribute to making AI safer?
