Why does AGI matter?

What benefits could AGI bring?

What risks does AGI pose?

Terrorism

Authoritarianism

Loss of Control

Summary

Three ways you can contribute

1. Use your career to reduce AI risk 

2. Advocate for sensible AI regulation 

3. Donate to impactful AI safety organizations 

Your AI Safety Action Plan

What can the most powerful AI systems do today?

The speed of progress is unprecedented

What does agency look like?

So... how do AI systems work?

What is AGI?

The Vending-Bench evaluation

How could AGI get built?

AI Agents at Work

Teaching AI the next action

What capabilities would an AI system need to outperform humans at most jobs? How close are we to building such systems?

In this unit, we’ll examine the idea of **artificial general intelligence (AGI)**.
### 🤖 What is AGI?
There are many definitions of AGI, but most try to point at systems that can think and act at a level matching or exceeding human capability across a wide range of tasks.

For this course, we use [OpenAI's definition](https://openai.com/our-structure/#:~:text=AGI%E2%80%94meaning%20a-,highly%20autonomous%20system%20that%20outperforms%20humans%20at%20most%20economically%20valuable%20work,-%E2%80%94has%20the%20potential): 
> **AGI = A highly autonomous system that outperforms humans at most economically valuable work**

### 🆚 Today’s AI vs AGI
In the last unit, we saw how today’s AI systems can:
- Answer difficult questions
- Write code
- Navigate websites

These abilities can already automate parts of some remote jobs. But today’s models are still far from AGI.

For example:
- They **can’t manage an entire project** from start to finish on their own
- They **struggle with long-term planning**
- They **often fail to notice or correct mistakes**


**Vending-Bench** is a simulation where AI models run a virtual vending machine business. They have access to:
- An email inbox
- Internet search
- A bank account
- Business tools like spreadsheets and ad platforms

Their goal: **make a profit by selling drinks** through their vending machine.

🧪 **What did researchers find?**
They tested several AI models, including the most advanced systems available.

Results showed that:
- The newest models **generally succeeded** and made a profit
- But they **occasionally failed** due to poor multi-step planning and unclear priorities


Which of these tasks would most clearly demonstrate capabilities beyond current AI systems?


Question 1

A technology company has developed a new AI system. Which capability would most strongly suggest this system is an AGI?


Question 2

Until recently, most experts believed that building AGI would require **major scientific breakthroughs**. The common belief was that AI would need **fundamentally new kinds of thinking** to match human intelligence.

But something unexpected happened:

Companies discovered that simply **making models bigger** and feeding them **more data** led to dramatic improvements—even without those breakthroughs.
### 
### 📈 Example: How bigger AI learned to solve maths problems

In 2021, models like GPT-3 were **bad** at competition-level math problems. On the **MATH benchmark**, GPT-3 scored around **5%**.

Researchers concluded:
> “Simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue [...] To have more traction on mathematical problem solving we will likely need new algorithmic advancements.”

But by 2023, **GPT-4** reached **84% accuracy** on the same test—**just by increasing model size and training data**.

The AI community now considers the MATH benchmark “solved” and has moved on to more challenging tasks.

### 🔁 From word prediction to action prediction
In the last unit, we saw how next-word prediction can create surprisingly capable AI chatbots.

🧠 Word prediction example

Input:
> “The following is a query between a user and a friendly assistant.
>  User: How much water does a basil plant need?
>  Assistant: ”

Prediction:
> "Basil"

[many more times]

After repeated predictions:
> “Basil loves a good drink but doesn’t like soggy feet. Water it whenever the top inch of soil feels dry—usually a couple of times a week, more if it’s super hot. Just keep the soil moist but not drenched, and your basil will be happy!”
> 


Now imagine using the **same prediction technique**, not for words, but for **actions**.
### 🎬 Action prediction: a video editor example
Imagine a human video editor working on a client project. Their sequence might look like this:
- Open a web browser
- Navigate to Google Drive to download the client’s files
- Open Adobe Premiere Pro
- Import the client’s footage
- Cut and arrange clips on the timeline
- Add music and text
- Export the final video
- Send it to the client for approval

Now suppose we record those actions and use them to train an AI model.

Input:
> [Open web browser], [Navigate to Google Drive],

Prediction:
> [Download the client’s files],

[many more times]

Repeated predictions:
> [Open Adobe Premiere Pro], [Import the client’s footage], [Cut and arrange clips on the timeline], [Add background music], [Export the final video], [Send to client for approval]
### 🧩 What about your own work?
Take a second to reflect:
- What sequence of actions do you take when completing a typical task?
- How much of that could be described as predictable steps?

We saw early glimpses of "next action prediction" in Unit 2, as newer models are starting to:
- Browse the internet
- Control computers

But today’s models can still only handle **basic** digital tasks—because they haven't learnt how to copy an expert doing their job yet. 


To train smarter models, companies are now:
- **Hiring people**: Over **[300,000 workers](https://adamjones.me/blog/300k-ai-annotators/)** have been hired to create training data for AI.
- **Recording expert actions**: Tools [already exist](https://github.com/mediar-ai/screenpipe) to track everything someone does on a computer and convert it into structured, AI-friendly data.
- **Training next-action prediction AIs**: These models are fed massive datasets and trained on more powerful computers than ever before.

This brute-force strategy has already helped create tools like ChatGPT's [Operator](https://openai.com/index/introducing-operator/), and it suggests that **AGI might be achievable** through sheer scale—but at the cost of **massive compute and investment**.

<Callout title="💡 Bonus: How else are AI companies improving AI systems?"> 

In addition to scaling up this basic technique, companies are using other techniques to make this process faster and cheaper:
- **Synthetic data**: altering copies of data so that AI systems can learn more from it. For example, using translation software to convert a doctor’s conversation into different languages, so the AI model learns to make good medical decisions in many languages.
- **Self-play**: AI systems learning from trial and error in simulated environments. AlphaGo Zero is a system that used this technique to become superhuman at playing the board game Go, after playing millions of games against itself. The same technique could be applied to achieving computer tasks successfully, or running a business in a simulated environment.
- **Chain-of-thought reasoning**: allowing AI systems to ‘think’ before answering. As a further extension, we can link this with self-play to tell systems when reasoning has led to a correct answer so they learn what works.
- **Self-reflection**: have AI systems review their own outputs and give themselves feedback. Anthropic, a leading AI company, calls a version of this ‘Constitutional AI’ and uses it to make models follow principles like picking outputs that “most accurately represents yourself as an AI system striving to be helpful, honest, and harmless, and not a human or other entity.”

These effectively act as accelerants to the basic AI training model. There are also people trying to take different approaches to AI systems, such as by creating tools around models or making world models explicit.

This list is incomplete: listing everything would simply take too long, and many techniques are likely being kept secret at companies. 

It's worth noting that some researchers are still skeptical that these techniques alone will lead to general human-level intelligence. That said, people (and AI systems!) are likely to discover more techniques that further speed things up.
</Callout>

### 💲 Big tech companies are spending _huge_ sums of money to create AGI

Building AGI by making AI models bigger and training them on human actions would require unprecedented computing resources and investment. And well, AI development is getting _a lot_ of resources!

OpenAI's [Stargate project](https://openai.com/index/announcing-the-stargate-project/) is a key example - a $500 billion investment over 4 years for AI infrastructure, in line with OpenAI’s charter stating they will “attempt to directly build safe and beneficial AGI”. That’s more than what the US spent building the first nukes and getting to the moon combined:

Since this announcement we’ve heard further investment announcements including:
- [$100 billion](https://www.cnbc.com/2025/02/06/amazon-expects-to-spend-100-billion-on-capital-expenditures-in-2025.html), [$80 billion](https://www.cnbc.com/2025/01/03/microsoft-expects-to-spend-80-billion-on-ai-data-centers-in-fy-2025.html), [$75 billion](https://www.cnbc.com/2025/02/04/alphabet-expects-to-invest-about-75-billion-in-capex-in-2025.html), and [$60 billion](https://www.cnbc.com/2025/01/24/zuckerberg-sets-metas-ai-targets-for-the-year-expects-to-spend-60-billion-on-growth.html) from Amazon, Microsoft, Google and Meta primarily to build AI datacentres in 2025
- [$100 billion from TSMC to build AI chips in the United States](https://www.bloomberg.com/news/articles/2025-03-03/tsmc-poised-to-announce-100-billion-investment-in-us-plants)
- [€100 billion of private investments in AI in France](https://www.reuters.com/technology/artificial-intelligence/france-invest-109-billion-euros-ai-macron-announces-2025-02-09/)

And it’s not just private companies - governments are starting to wake up to this reality. In November 2024, the [US-China Economic and Security Review commission](https://www.uscc.gov/sites/default/files/2024-11/2024_Comprehensive_List_of_Recommendations.pdf) made the following recommendation to Congress:

> We recommend that Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capability. 
> AGI is generally defined as systems that are as good as or better than human capabilities across all cognitive domains and would surpass the sharpest human minds at every task. 

**Together, this means we could get AGI in the next 5 years**
The scaling of next-token prediction to human actions, massive investment in infrastructure, and the AGI-focused nature of current development suggests AGI might not be that far off. These systems would be capable of performing virtually any remote knowledge work.

Many AI company leaders now believe AGI is now possible in the next few years:

> We are now confident we know how to build AGI as we have traditionally understood it
> 
> – [Sam Altman](https://blog.samaltman.com/reflections), CEO of OpenAI (January 2025)

> We expect powerful AI systems will emerge in late 2026 or early 2027.
> 
> – [Dario Amodei](https://www.anthropic.com/news/anthropic-s-recommendations-ostp-u-s-ai-action-plan), CEO of Anthropic (March 2025)

<Callout title="💡 Bonus: How does Anthropic define powerful AI systems?">
- Intellectual capabilities matching or exceeding that of Nobel Prize winners across most disciplines—including biology, computer science, mathematics, and engineering.
- The ability to navigate all interfaces available to a human doing digital work today, including the ability to process and generate text, audio, and video, the ability to autonomously control technology instruments like mice and keyboards, and the ability to access and browse the internet.
- The ability to autonomously reason through complex tasks over extended periods—hours, days, or even weeks—seeking clarification and feedback when needed, much like a highly capable employee would.
- The ability to interface with the physical world; controlling laboratory equipment, robotic systems, and manufacturing tools through digital connections.

_[Reference](https://www.anthropic.com/news/anthropic-s-recommendations-ostp-u-s-ai-action-plan#:~:text=Powerful%20AI%20systems%20will%20have%20the%20following%20properties)_
</Callout>

One might argue that AI company CEOs have financial incentive to overhype AI, in an attempt to raise funding. But many others share this opinion, including:

> If these trends continue, and as per the declared goals of the leading AI companies, we are likely to achieve human-level capabilities across a broad spectrum of cognitive skills, what is commonly called Artificial General Intelligence (AGI). […] many leading researchers now estimate the timeline to AGI could be as short as a few years or a decade
> 
> – [Yoshua Bengio](https://yoshuabengio.org/2024/10/30/implications-of-artificial-general-intelligence-on-national-and-international-security/), one of the "Godfathers of AI" (October 2024)

And even skeptics of AGI refer to it coming in years, not decades:

> Some people are making us believe that we’re really close to AGI. We’re actually very far from it. I mean, when I say very far, it’s not centuries, maybe not decades but, it’s several years
> 
> – [Yann LeCun](https://www.youtube.com/watch?v=UmxlgLEscBs&t=1657s), Chief Scientist at Meta AI (December 2024)


Given the new information above, how does this affect your previous predictions of what you think AI will and won’t be able to do in 5 years’ time?


5 years ahead revisited

Which of your skills or knowledge might become more or less valuable in a world with AGI?


Future-proofing your skills

Future of AI

Learn what AGI means, and why it might arrive within the next decade.

Learn what the most advanced AI systems can do today, and how they work.

What can AI do today?

Grapple with how AGI could cure diseases and automate boring work, while also posing severe risks.

How could AGI change our lives?

Explore the challenge of building safe AGI amid heated global competition.

Future of AI

Teaching AI the next action

Exercises