What drives AI progress?

What can increasingly capable AI systems do?

How could progress accelerate from here?

Pathways to Harm

Loss of Control

Authoritarianism

Biological Misuse

Bring the arguments together

What’s at stake?

What future do you want?

Competition and coordination problems

Responding to the Hugging Face incident

Orienting to the strategy

The technical challenge

Following the transition

Pacing and alternative strategies

Examining the objections

Your assessment of the situation 

Choose your focus

Go deep in one area

Make a plan

Create your 1-pager (optional)

Optional advice

Next steps: Deeper-dive courses

Next steps: Programs

“What are your timelines?” is a common question in AI safety circles and beyond. People usually mean when you expect AGI to arrive, though what counts as AGI is often left vague.

To understand how AI could transform civilization, for better or worse, we need to be more specific about the capabilities we care about. There are several milestones worth tracking, each with different implications for how AI could change the world:

- **Matching skilled humans at specific tasks**, such as fixing software bugs or solving mathematics problems.
- **Completing longer tasks without human help.** [METR’s time horizons](https://metr.org/time-horizons/) measure task length using the time a human expert would need. A one-hour horizon at 50% reliability means an AI is predicted to succeed half the time on tested software tasks that take human experts about an hour.
- **Automating all economically valuable work:** performing those tasks at least as well and as cheaply as humans.
- **Being capable of permanently disempowering humanity:** being able to take control away from humans if the systems attempted to, despite efforts to stop them.
- **Automating AI research and development:** doing the work needed to build more capable and general AI systems, potentially before AI can automate many other jobs.
- And many more...

When thinking about AI progress, it is helpful to distinguish several questions:

- **What resources and methods are improving?** This may include compute, data, algorithms, training methods and the ways systems are deployed. 
- **What can systems actually do?** Which tasks they can complete, how reliably, with how much human help and at what cost.
- **What would those capabilities enable - what is the impact if adopted at scale?**
- **Could AI change the process of improving AI?** AI research assistance might develop into a feedback loop in which better and better systems help or at some point fully build their successors.

Some readings were written in 2025, and AI moves fast! Use the newer evidence and forecasting updates to assess if those arguments stood the test of time. 


This chunk brings together evidence about the current state of AI capabilities and the rate of AI progress. It should give you a good sense of what tasks AIs can complete autonomously. 

If you are newer to interacting with AI systems, you should - beyond only reading about it here - try to push the models to their limits yourself. Maybe pick a task you dislike doing and see how much of it you can hand over to AI. To get started, we recommend setting up [Claude Code](https://code.claude.com/docs/en/desktop-quickstart) or [Codex](https://developers.openai.com/codex/quickstart) if you haven't yet. 

And then in the readings you will explore AI performance on tightly defined benchmark tasks as well as on more open-ended/qualitative work. 

You should practice extrapolating these trends, especially the ones that appear non-linear, and consider different rates of progress and what bottlenecks might still be ahead. Also compare how quickly AI is improving at different kinds of tasks and think about what could explain the differences.

And if you have extra time, we recommend you try your hand at making some predictions on, for example, [Metaculus’s AI 2027 questions](https://www.metaculus.com/tournament/ai-2027/). 


> Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion.'

— [Irving J Good, 1965](https://www.historyofinformation.com/detail.php?id=2142)

To open with yet another common AI safety question: **will we soon witness an intelligence explosion - a period of rapid, self-reinforcing growth in AI capabilities, as AI systems increasingly drive their own improvement?**

This is the possibility [I. J. Good described in 1965](https://scienceplusplus.org/metascience/assets/Good1965.pdf), and it seems increasingly plausible.

To lay out a rather oversimplified argument:
- Current LLM-based AI systems are quite good at coding.
- They are getting better at research and generally finding solutions to problems, especially in domains where results can be checked automatically (software, math, etc).
- [Reinforcement learning](https://arxiv.org/abs/2501.12948) - rewarding systems for successful behavior - has become the important driver of these improvements.
- AI systems are already being used to develop future AI systems.
- In August 2026, [Anthropic estimated that Claude was “leading” 26% of its measured AI R&D work](https://www.anthropic.com/institute/measuring-pace-of-ai-development), up from under 1% in February. Though, “leading” here means doing most of a task with human supervision, rather than fully owning the process.
- A naive extrapolation suggests that Claude could lead most of this work during 2027, though the remaining tasks may be harder, and extending a short trend is highly uncertain.
- At the same time AI systems still struggle with [research taste](https://www.anthropic.com/institute/recursive-self-improvement), i.e. choosing worthwhile questions, interpreting results and knowing when to abandon/double-down on an approach.
- This might improve too, but whether - and how soon - it will be solved is less clear.
- Even without solving it completely, automating more of the surrounding work will very likely shorten development cycles.
- AI systems can then also build tools that help them accomplish more and improve their pipelines.
- Particularly important are [improvements to algorithms](https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/): getting more capability from the same resources.
- Better AI could therefore accelerate AI research, producing better AI, which accelerates research further and so on.
- If this feedback loop becomes powerful enough, it could lead to superintelligence in the [Bostromian definition](https://nickbostrom.com/papers/survey.pdf), “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.”
- But feedback alone does not guarantee this happening. What matters is how much each round improves capabilities, how much it needs in terms of resources and [how long that round takes](https://arxiv.org/abs/2608.14426).
- Though, it seems clear to this author that at some point in the future - perhaps not soon and not scaling these exact systems - we will solve this problem insofar as a sufficiently valuable and clearly-specified goal gets reached eventually (many people do strongly disagree about the merits of [such techno-determinism](https://publicism.info/philosophy/superintelligence/15.html)!)
- Currently there may be significant constraints such as available compute, useful data/environments, reliable feedback, experiment duration and generally diminishing returns.
- Compute, data and algorithms are useful inputs to keep in mind
- Please explore some assumptions about compute, algorithms and automation in [Epoch AI’s GATE simulator](https://epoch.ai/gate) - this is great to test/build your intuitions. 
- Other trajectories remain possible too - we could see more gradual improvement, periods of stagnation, or sudden breakthroughs.
- Nor does everything need to depend on scaling LLMs. Brain-inspired approaches or a relatively small “seed AI” capable of improving itself are [other proposed possibilities](https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement), though their feasibility remains uncertain.

To recap, the central question of this chunk (and maybe humanity's future) is how much better AI systems can accelerate the development of their successors, and whether/when that acceleration strengthens/speeds up into an intelligence explosion. 

As you read AI 2027 and their 2026 update, look for the assumptions their account makes about the parameters described above, research taste, available resources and the time it takes to turn discoveries into more capable systems.


Think about your job, studies, research, volunteering or another substantial project. Choose an area of work you know well.

1. **What does it involve?** 
    1. Identify three to five important activities, including any that require judgment, coordination or relationships with other people.
2. **What could AI take responsibility for a year from now?** 
    1. Which activities could it complete reliably, and what human involvement would remain? Make one concrete prediction and say how confident you are.
3. **Why do you expect this?** 
    1. Draw on at least two pieces of evidence from the unit. Identify the most important improvement needed, the biggest remaining constraint, and an observation that would change your mind.
4. **What if AI became substantially more capable, faster and cheaper?** 
    1. How would that change your role - and how your workplace, university or organization operates? What would remain difficult?


AGI Strategy

What drives AI progress?

Resources (50 mins)