What drives AI progress?

What can increasingly capable AI systems do?

How could progress accelerate from here?

Pathways to Harm

Loss of Control

Authoritarianism

Biological Misuse

Bring the arguments together

What’s at stake?

What future do you want?

Competition and coordination problems

Responding to the Hugging Face incident

Orienting to the strategy

The technical challenge

Following the transition

Pacing and alternative strategies

Examining the objections

Your assessment of the situation 

Choose your focus

Go deep in one area

Make a plan

Create your 1-pager (optional)

Optional advice

Next steps: Deeper-dive courses

Next steps: Programs

**What would it take to secure a safe and broadly beneficial future as AI becomes more capable than humans?**

In Unit 2, you looked at how fast AI is developing. In Unit 3, you investigated how increasingly capable systems could cause catastrophic harm. In this unit, you will look at some of the strategies for addressing these risks.

A [good strategy](https://archive.ph/TuB5G) will need to have the following: 
1. An objective/target
    1. What outcome are we trying to achieve and for whom? 
2. A diagnosis
    1. What obstacles/dynamics/walls are in our way?
3. A theory of success/winning 
    1. Why can and should our chosen approach overcome these obstacles?
4. A feasible course of action
    1. Who needs to do what, using what resources and under what conditions/incentives? What choices and trade-offs does getting the work to happen involve?
5. The ability to adapt to change 
    1. What assumptions does our strategy depend on and what evidence or changes in the world would require us to revise it?

We will be using the AI Futures Project’s **Plan A from AI 2040** as our central case study. 

By the end of this unit, you should be able to:
- Explain arguments for why their strategy makes the future safer than it would otherwise be.
- Assess its assumptions about AI progress, incentives, and adaptability to political change.
- Evaluate its effects on catastrophic risk and the distribution/concentration of benefits and power.
- Decide (provisionally) whether to endorse it and what changes could earn them your support.


AI companies have invested some resources in the safe training and deployment of their current systems (prosaic alignment), with hopes to scale safety practices to AGI and superintelligence. However, the hard technical problem of reliably aligning powerful AI with human values remains unsolved. 

Some researchers think current approaches cannot get us to safe AGI. Others think we could get there, or at least close enough to get the AIs themselves to do meaningful safety work, by improving training and combining layers of oversight with safeguards that catch failures.

Even if these approaches can work in theory, companies might underinvest in them in practice. Failures in current systems like the Hugging Face attacks discussed in Unit 1 show that alignment and oversight are far from on track, and rapid capability advances could leave less time to address them. 

Worse yet, important trends and techniques seem to be declining in effectiveness against increasing capabilities - [monitorability](https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-do-a-concerning-amount-with-no-chain-of-thought) of logs (be they output or chain-of-thought), evaluation awareness, our increasing reliance on [AIs themselves](https://x.com/RyanGreenblatt/status/2092692685224325542) for this work and emerging power-seeking/deceptive behavior have made people fear we're falling behind in the "fix it one step at a time" approach to alignment. 

The readings examine how labs plan to make AGI safe, some of the empirical results they're getting with existing alignment practices, and how they think about scaling these up. 


> We live during the hinge of history. Given the scientific and technological discoveries of the last two centuries, the world has never changed as fast. We shall soon have even greater powers to transform, not only our surroundings, but ourselves and our successors. If we act wisely in the next few centuries, humanity will survive its most dangerous and decisive period.

[Derek Parfit](https://www.globalprioritiesinstitute.org/wp-content/uploads/William-MacAskill_Are-we-living-at-the-hinge-of-history.pdf#page=2)

The AI Futures Project's Plan A offers one proposal for what acting wisely during this transition could look like. Whether or not achieving it as laid out is desirable is a different question. 


Let’s start this chunk with a quick thought experiment. **What would have to be true for you to be comfortable with AI progress continuing at full steam** - all the way to superintelligence as described earlier in the course? 

Spend a few minutes thinking about the characteristics of that world - what has to be true? 

That might be something like more capable AIs would bring enormous, well-distributed benefits, we could reliably keep them aligned as they became more capable, and we could spot serious problems before they caused catastrophic harm. We could go in and fix mistakes as we went. In that world, slowing down would likely mostly mean delaying good things.

But this does not appear to be the world we live in. AIs already don’t reliably do what we want. They are becoming more capable quickly, while our methods for aligning them and checking whether they are actually aligned are not on track - and there is no obvious and easy way to get them to be. 

Furthermore, we cannot assume we will always be able to muddle through and fix mistakes after they happen, as with full loss of control you only get one shot at goal. 

The labs do have safety plans (they vary in quality/seriousness), but it remains unclear whether those plans will be enough as capabilities continue their advance. And the US government seems no longer fully asleep at the wheel, but one can wonder whether it is heading in the right direction. 

With companies and countries under immense, sometimes imagined pressure to stay ahead, we should not assume the race will slow itself down enough even when that would be the most sensible thing to do.

So what could we do in this world? This is where pacing the frontier - the theme of this chunk - comes in. 

The basic pitch for it is: Let's slow capability development enough to give alignment research, other safety/defense work, and our institutions a chance to catch up. We could then use that time to improve our alignment techniques, build agreements and institutions that can enforce limits, and prepare to reap and distribute the benefits of advanced AI.

That only helps if the extra time actually improves our position. Coordinating the right kind of slowdown looks to be rather tricky, and delaying useful AI would have real costs. A slowdown could also backfire, for example through compute overhang or a lack of AI systems capable enough to help with alignment research. For these reasons, some people prefer an indefinite ban on AI development aimed at RSI and ASI. 

But if slowing down sufficiently reduces the risk of catastrophic harm or permanently losing control, such costs could be worth paying. As you read, please keep track of your disagreements and uncertainties with this line of thinking.


Some objections to Plan A accept the authors’ framework but argue that another plan would be better. Others question the framework, whether that be the desirability of the proposed end state or the assumptions it makes about compute stock, competition, and cooperation. This section examines the most developed of these objections.


Prepare brief notes for the discussion. A few sentences for each prompt are enough.

1. **Technical dependency:** Identify one technical problem whose resolution matters to Plan A. What evidence would increase your confidence that it had been addressed?
2. **Political dependency:** Identify one commitment Plan A requires from an institution/actor. Why might they make it, and why might it fail?
3. **Connection to Unit 2:** Which judgment about future AI capabilities or the pace of progress most affects your recommendation?
4. **Your recommendation:** Name one action you currently support and why it would reduce catastrophic risk.



AGI Strategy

Examining the objections

Resources (35 mins)