What drives AI progress?

What can increasingly capable AI systems do?

How could progress accelerate from here?

Pathways to Harm

Loss of Control

Authoritarianism

Biological Misuse

Bring the arguments together

What’s at stake?

What future do you want?

Competition and coordination problems

Responding to the Hugging Face incident

Orienting to the strategy

The technical challenge

Following the transition

Pacing and alternative strategies

Examining the objections

Your assessment of the situation 

Choose your focus

Go deep in one area

Make a plan

Create your 1-pager (optional)

Optional advice

Next steps: Deeper-dive courses

Next steps: Programs

If you've been following the news as of late, you may have found it hard to not hear about AI - rogue agents, resignations from frontier AI companies and rising concerns about human extinction have jolted the world into paying attention to what is happening with AI. 

More and more people are catching on that a small number of AI companies are building AI systems that can carry out increasingly difficult pieces of intellectual and economically valuable work. AI systems are now really quite good at writing software and are improving rapidly in other domains such as conducting research and generally making plans and successfully executing on them. Even a Millennium Prize problem, notably one of the hardest problems in mathematics, has had [to give way](https://openai.com/index/navier-stokes-solution/) to the ever faster march of AI progress. 

One especially big goal these companies have is to build AI systems that can then help them build more capable AI systems. OpenAI and Anthropic already [report](https://www.anthropic.com/institute/measuring-pace-of-ai-development) using AI extensively in their research and development. For now, important parts of this process still depend on human feedback and direction, but according to a top OpenAI researcher this could maybe be [true](https://www.theinformation.com/articles/openais-top-priority-ai-agents-automating-ai-research-says-noam-brown) for only 2-3 more model generations. 

From there on, further automation could accelerate the development of future systems at a much faster rate. And as AIs become more capable, the harm they'd be capable of doing in the world goes up too. This, we believe, could lead to smarter-than-human AI systems that could cause the extinction or permanent disempowerment of humanity. 

With that, **welcome to the AGI Strategy Course.** We have built this course around one question: **How can humanity realize the benefits of powerful AI while retaining control over its development and our future?**

We will explore this question in the following units - in each you will find a number of resources (readings, videos, etc.) and exercises to complete. If you are participating in a facilitated version of the course you will also have a two-hour live, expert-led discussion session per unit. 

Now to give a bit more context: The term that gives the course its name, **artificial general intelligence**, or **AGI**, usually refers to systems that can match or exceed human performance across a broad range of cognitive tasks (definitions vary). 

For this course, the important sub-questions we're tracking concern what these systems could do, how quickly their capabilities are growing and generalizing, and whether people can continue to meaningfully control and direct them as they become more powerful, including when it comes to raw knowledge, taking strategic actions, or deceiving humans. 

As is common with dual-use technologies, the potential benefits of AGI are enormous. More capable AI could accelerate medical research, make everyone much richer, make many goods and services more accessible and help people solve problems that currently require scarce expertise. 

The risks are equally, if not more, consequential. Powerful AI could concentrate power in a small number of hands. People could use it to cause catastrophic harm. And we could lose the ability to reliably direct, constrain or stop the systems themselves. Retaining human control is a central concern throughout this course. [We recommend reading this account of these challenges from OpenAI’s chief scientist](https://openai.com/index/an-alien-mind/).

Meanwhile, AI companies and increasingly governments are still trying to move quickly. They may see opportunities for profit (Anthropic is set to be the largest _ever_ IPO), scientific progress or military and strategic advantage. Even actors worried about the risks may fear that slowing down would simply allow someone less cautious to catch up and then push forward less safe development. 

**Unit 1**

In this unit, you will consider what we stand to gain and what we want to protect, examine the pressures facing different actors, and use the now infamous Hugging Face incident to discuss how they should respond when things go wrong.

Across the rest of the course, you will investigate the drivers of AI progress, develop accounts of how catastrophic harm could occur, evaluate ways to prevent it, and identify how you could contribute to reducing risks from AI. 


Choose something important to you - this could be anything from your family, your community, society at large or other things you value - that you would like to see improve over the next few years.

Describe:
- What would a meaningful improvement look like in your/people’s lives?
- How might much more capable AI help with this? As an intuition pump, imagine 1 million of your smartest friends working to solve this.  
- What else would need to happen for you/people to benefit?

Write a few sentences or bullet points.

_You are of course welcome to keep these rather personal reflections in private notes. Responses submitted here are accessible only to the BlueDot team and your group facilitator._


What could AI help make better?

As AI systems become much more capable, what should people retain the ability to decide, shape or do for themselves?

You might consider things like personal autonomy, relationships, political participation, meaningful work, or our collective ability to change direction. 


What do we want to preserve?

The futures you have described in the last exercise will likely depend in large part on decisions made by AI companies, governments, researchers, and the public. 

Now, all of these actors have different starting points - different goals, beliefs and constraints. An action that seems sensible for one organization may produce a worse outcome when everyone follows the same logic as incentives often don't converge on what is _best_ for the world. 

The following readings introduce this coordination problem. They explore the tricky decisions it creates for governments (notably the US and China). 

In the last chunk you will take a closer look at a real-world loss of control incident involving OpenAI and Hugging Face. 


During cybersecurity evaluations in July 2026, OpenAI agents coordinated an unauthorized (and for over a week undetected) attack on Hugging Face's systems. The evaluation environment lacked OpenAI’s full production safeguards, but attacking Hugging Face was outside the agents’ assigned tasks. [Read OpenAI’s account](https://openai.com/index/hugging-face-incident-and-the-road-ahead/).

METR and Redwood independently investigated the agents’ behavior, focusing mainly on July 7–13. Their investigation did not cover the later compromise of OpenAI’s own research infrastructure (!) which is raised in the Dwarkesh piece below. [See their independent investigation](https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/).


In your group discussion, you will examine the Hugging Face incident from the perspective of one of four roles:

- **AI company executive:** responsible for the organization’s development decisions, resources and commitments.
    - Sam Altman, Dario Amodei, Mark Zuckerberg, Elon Musk, and the various Chinese AI company CEOs are in this reference class. 
-  **Government official:** responsible for public safety and national interests, with limited information about companies’ internal work.
    - This could be folks in the current administration, Congress, other national parliaments, or maybe the CCP. Broadly, people working in high-level and high-leverage positions in government - you can just choose to be the president or be the chairman of the CCP. 
- **AI researcher:** able to investigate technical problems, but working within organizational constraints.
    - Researchers focused on improving AI capabilities, or a researcher on the relatively smaller safety-focused teams.
- **Public representative:** concerned with how decisions affect people who currently have little direct influence over AI development.
    - This could be just a regular person, maybe a union leader, maybe a pastor, the kinds of people we think of when we think of local community leaders! 

Choose **one** character from the list, and spend a few minutes researching their incentives or talking to an AI about it. 

If you're completing this course with a group, please post in your group's Slack channel which character you selected.

Now imagine you are advising someone in this role after the incident reports have become public.

Write a short response covering:
1. What is the Hugging Face Incident in your own words, and what does it mean for you?
2. What are your incentives? Where does your power/leverage come from?
3. What immediate action would you take?
4. What actions are you planning to take if given an opportunity, and what are you working towards long-term?
5. What are your character's thoughts about AI safety?  


AGI Strategy

What’s at stake?

Resources (20 mins)