Why does AGI matter?

What benefits could AGI bring?

What risks does AGI pose?

Terrorism

Authoritarianism

Loss of Control

Summary

You can make a difference

How our courses work

Getting started

What's already possible

The pace of progress

From tools to agents

What is AGI?

Vending-Bench: measuring progress towards AGI

Building AGI and beyond

The huge bet on AGI

What capabilities would an AI system need to outperform humans at most jobs? How close are we to building such systems?

In this unit, we’ll examine the idea of **artificial general intelligence (AGI)**.
### 
### **What is AGI?**
There are many definitions of AGI, but most try to point at systems that can think and act at a level matching or exceeding human capability across a wide range of tasks.

One of the simplest is the [definition from OpenAI's Charter](https://arc.net/l/quote/iosyuqvg):
> **AGI = A highly autonomous system that outperforms humans at most economically valuable work**

### **Today's AI is approaching AGI**
In the last unit, we saw how today’s AI systems can:
- Answer difficult questions
- Control computers
- Complete increasingly complex coding tasks

These abilities can already automate parts of some remote jobs, and it's becoming harder to distinguish current AI systems from AGI. Some AI experts argue that [we're already there](https://x.com/deanwball/status/2001068539990696422).
### 
### **But it can't do the whole job yet**
Frontier AI systems still struggle in a few ways that are important for economically valuable work. ‘Jaggedness’ is the idea that AI models can be very good at tasks humans find hard, whilst also being surprisingly bad at some tasks humans find easy.

Historically, AI models haven’t been able to [count the number of Rs in the word ‘strawberry’](https://techcrunch.com/2024/08/27/why-ai-cant-spell-strawberry/), or determine that [9.9 is bigger than 9.11](https://www.youtube.com/watch?v=vB9dJt9j-5M), although these quirks are becoming harder to find as capabilities improve.

In particular, today’s AI models struggle with:
- **Reliability** on longer and more open-ended tasks (although this is improving as the graph in unit 1.2 showed)
- **Interacting with the physical world** (although this isn’t necessary to automate remote work)



Where the METR graph measures the length of tasks AI models can do, **Vending-Bench** tries to directly measure AI models' ability to autonomously do economically valuable work.

Vending-Bench is a simulation where AI models run a vending machine business over a year and scored on their bank account balance at the end. They start with:
- $500
- An email inbox
- Internet search
- A bank account
- Business tools like spreadsheets and ad platforms

Over a year ago, some of the frontier AI models could turn a profit, but most lost money. Today, all of the frontier models make a large profit. **The current best, Anthropic's Claude Opus 4.7, turned it's original $500 into over $10,000.**

<Embed url="/images/content/foai/vending-bench-results.webp"/> 
Source: [Andon Labs](https://andonlabs.com/evals/vending-bench-2)


Three main concepts explain how today's frontier AI gets built.
- Scaling: bigger models, more data
- Reinforcement learning: practice and feedback
- Recursive self-improvement: AI helping improve AI
### 
### Bigger models, more data
Until recently, most experts thought building AGI would require major scientific breakthroughs. The main algorithm behind large language models is trained to be very good at predicting the next word in a sentence (kind of like a fancy autocomplete). The common belief was that AI would need fundamentally new kinds of thinking to match human intelligence.

What actually happened was simpler. AI labs found that making these ‘next word predictor’ AI models bigger and feeding them more data led to dramatic improvements across a range of domains.

<Callout title="Example: How bigger AI learned to solve maths problems">
In 2021, GPT-3 scored around 5% on the [MATH benchmark](https://arxiv.org/abs/2103.03874), a set of competition-level maths problems. The researchers behind it concluded:

> "Simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue [...] To have more traction on mathematical problem solving we will likely need new algorithmic advancements."

By 2023, GPT-4 hit 84% on the same test, just by training a bigger model on more data. The AI community now considers MATH solved and has moved on to harder benchmarks.
</Callout>
### Practice and feedback
Scale alone doesn't make a model good at certain specific tasks. For that, the model needs practice. 

After the basic training, AI labs let the models practice specific tasks and give them feedback on whether they did well or poorly. When the model succeeds, the steps that led to success are reinforced. When it fails, those steps become less likely. Run this for billions of attempts and the model gets reliably good at that kind of task. 

This approach works best where the answer can be checked automatically such as maths with known answers, code that can be run to see if it works or not, or other tasks with clear completion criteria.

This stage is called reinforcement learning. It's the reason today's reasoning models can think through a hard problem before answering, and it's how agents like [Claude Code](https://claude.com/claude-code) can take a software task and work on it for hours.
<Callout title="What forms does this kind of practice and feedback take?">
A few of the techniques labs use:
- **Chain-of-thought reasoning.** Models are given space to "think" by writing out intermediate steps before answering. Combined with feedback on which lines of reasoning lead to correct answers, the model learns better thinking strategies over time.
- **Self-play.** Systems learning from trial and error in simulated environments. AlphaGo Zero became superhuman at Go by playing millions of games against itself. The same idea is being extended to code and reasoning, where models can generate their own training problems and then try to solve them
- **Constitutional AI.** Anthropic's approach: instead of humans ranking outputs, an AI model evaluates outputs against written principles (like staying helpful, honest, and harmless) and provides the feedback signal. Cheaper and more scalable than human-only ranking.
</Callout>
### AI helping improve AI
The newest twist is that AI itself is starting to do the work. Frontier labs report that models are now writing meaningful portions of training code, designing experiments, and helping interpret results. Each generation of more capable AI helps build the next generation faster than the last did.

We may reach a point where AI models are able to train the next generation of AI without any human input at all. In fact, this is the explicit goal of many of the AI companies. If that happens, how far beyond human intelligence could these models go, and what might it mean for humans?


### **Tech companies are spending _huge_ sums of money to create AGI**
Training bigger AI models, improving them with reinforcement learning and then using those models to train better AI takes unprecedented computing resources and investment.

Total spending on data centres is approaching $1 trillion dollars, more than the Manhattan Project, Apollo Program and International Space Station combined (even adjusting for inflation).

<Embed url="/images/content/foai/ai-capex-vs-megaprojects.webp"/>
Source: [Fin Moorhouse](https://x.com/finmoorhouse/status/2044933442236776794)

The only reason this level of investment makes sense is because AI companies and their investors truly believe that their AI models will be able replace large fractions of the economic value produced by humans today.

And it’s not just private companies. Governments are starting to wake up to the strategic importance of AGI. As far back as November 2024, the [US-China Economic and Security Review commission](https://www.uscc.gov/sites/default/files/2024-11/2024_Comprehensive_List_of_Recommendations.pdf) made the following recommendation to Congress:

> We recommend that Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capability. 

### Many AI leaders expect we will build AGI within the next 5 years

> We are now confident we know how to build AGI as we have traditionally understood it
> 
> – [Sam Altman](https://blog.samaltman.com/reflections), CEO of OpenAI (January 2025)

> We expect powerful AI systems will emerge in late 2026 or early 2027.
> 
> – [Dario Amodei](https://www.anthropic.com/news/anthropic-s-recommendations-ostp-u-s-ai-action-plan), CEO of Anthropic (March 2025)

<Callout title="💡 Bonus: How does Anthropic define powerful AI systems?">
- Intellectual capabilities matching or exceeding that of Nobel Prize winners across most disciplines—including biology, computer science, mathematics, and engineering.
- The ability to navigate all interfaces available to a human doing digital work today, including the ability to process and generate text, audio, and video, the ability to autonomously control technology instruments like mice and keyboards, and the ability to access and browse the internet.
- The ability to autonomously reason through complex tasks over extended periods—hours, days, or even weeks—seeking clarification and feedback when needed, much like a highly capable employee would.
- The ability to interface with the physical world; controlling laboratory equipment, robotic systems, and manufacturing tools through digital connections.

_[Reference](https://www.anthropic.com/news/anthropic-s-recommendations-ostp-u-s-ai-action-plan#:~:text=Powerful%20AI%20systems%20will%20have%20the%20following%20properties)_
</Callout>

One might argue that AI company CEOs have financial incentive to overhype AI, in an attempt to raise funding. But many others share this opinion, including:

> If these trends continue, and as per the declared goals of the leading AI companies, we are likely to achieve human-level capabilities across a broad spectrum of cognitive skills, what is commonly called Artificial General Intelligence (AGI). […] many leading researchers now estimate the timeline to AGI could be as short as a few years or a decade
> 
> – [Yoshua Bengio](https://yoshuabengio.org/2024/10/30/implications-of-artificial-general-intelligence-on-national-and-international-security/), academic and one of the "Godfathers of AI" (October 2024)

And even skeptics of AGI refer to it coming in years, not decades:

> Some people are making us believe that we’re really close to AGI. We’re actually very far from it. I mean, when I say very far, it’s not centuries, maybe not decades but, it’s several years
> 
> – [Yann LeCun](https://www.youtube.com/watch?v=UmxlgLEscBs&t=1657s), Former Chief Scientist at Meta AI (December 2024)


Really try to internalise what the trajectory of this technology means. What's one thing about this you find most exciting, and one thing you find most concerning?


Future of AI

The huge bet on AGI

Exercises