Why does AGI matter?

What benefits could AGI bring?

What risks does AGI pose?

Terrorism

Authoritarianism

Loss of Control

Summary

You can make a difference

How our courses work

Getting started

What's already possible

The pace of progress

From tools to agents

What is AGI?

Vending-Bench: measuring progress towards AGI

Building AGI and beyond

The huge bet on AGI

Throughout human history a handful of technologies have reshaped civilization: agriculture, fossil fuels, nuclear fission. Each brought with it vast benefits and serious risks. In fission's case, the risks were even existential. 

We think AGI will be a bigger deal than all of them.

That is because AGI would automate the very thing that put humans in charge: intelligence. Our ability to plan, solve problems and adapt creatively is what sets us apart from other animals. It is why humans, and not, say, gorillas, run the planet, and why the gorillas' survival now depends on our choices rather than theirs.

Every earlier technology was a tool that amplified the power of human intelligence. AGI could match or exceed what makes us special. That is why many liken its development to the arrival of a new species. 

There is also little reason to think human intelligence is the ceiling. AI systems could eventually become far smarter than all of humanity combined, reaching what is often referred to as superintelligence.

Trivially, the more capable AI becomes, the more good and harm it can do. If we get it wrong along the way humanity could be permanently disempowered or even wiped out. But if we get it right, AGI could usher in an era of growth, abundance and opportunity unlike anything in human history.


This unit covers both the benefits and the risks of AGI.


AGI could be the most powerful tool humanity has ever built. The potential upside is extraordinary, but it's worth being specific about why. Let's take a look: 

### Transforming scientific discovery

Many believe science is bottlenecked by human attention. Even brilliant researchers can only read so many papers, run so many experiments, and hold so many ideas in their head at once. The biggest breakthroughs - CRISPR, mRNA vaccines, penicillin - often came from connections across fields. Those connections are hard to find when no one can keep up with the literature in even a single discipline.

AI is already starting to change this. AlphaFold solved the protein-folding problem, a challenge that had stumped biologists for 50 years. AI systems have discovered new antibiotics that work against drug-resistant bacteria. Google's diagnostic tools are matching or outperforming specialists in detecting certain cancers.

AGI could take this much further by not just analyzing data, but further designing experiments, generating hypotheses, and connecting ideas across every field of science simultaneously. 

Some researchers describe this as the "compressed 21st century": decades of progress packed into years delivered to us by "a country of genius in a datacenter."

Now this of course does not mean physics suddenly stops applying. Experiments will still take time. Clinical trials still need to run. Real-world adoption still has to happen. But the rate-limiting step shifts from "can we figure this out?" to "can we build and test it fast enough?", which is a very different kind of problem and much more like engineering than science. 

### Economic growth

The economic implications follow from the science - and from broader automation. If you can dramatically accelerate R&D, you can discover new materials, new energy sources, new manufacturing methods. Productivity gains compound.

For context: at the roughly 2% annual growth most developed economies sustain, living standards double every 35 years. Some economists argue AGI could push growth to more than 10-20% per year - potentially doubling living standards every 5. 

In such a world goods and services that are currently scarce could become broadly accessible, similar to the way smartphones went from luxury items to being used by billions in under two decades.

Even the more conservative estimates are striking. And AI is already generating measurable productivity gains across industries - and current AI systems are far less capable than what we are picturing when talking about AGI.  

### There is a catch

None of this happens by default. Every technology on this list - from agriculture to nuclear energy - produced enormous benefits and enormous harms, depending on who controlled it and how it was deployed. AGI is no different, except that the stakes and the speed are both higher, and it is up to us to get this right. 

<Callout title="💡 Bonus: What does the future of human labour look like?">
AI won't just automate routine tasks; it will increasingly perform work once thought uniquely human - creative, analytical, and interpersonal. Within a single generation, we may face a profound change in how our economy works.

As AGI develops, even careers we consider secure - doctors, lawyers, artists, scientists - could face disruption, alongside manual trades. This shift might lead not just to temporary unemployment but to a lasting disconnect between human skills and economic value for many people.

The benefits of AGI might flow disproportionately to those who own and control these technologies, potentially creating unprecedented concentrations of wealth. This economic power often translates to political influence, which could further widen these gaps.

Throughout history, our economic systems have distributed resources largely based on people's contributions through work. What happens in a world where many people's labour might no longer have significant market value? This raises profound questions about how society should function.

This possible future challenges us to rethink not only economics but also human identity and purpose. We may need new economic models, thoughtful policies for more equitable distribution of AI's benefits, and perhaps a fundamental shift in how we value human contributions beyond paid work.

While these challenges are serious, they also create an opportunity to build a society that places greater emphasis on human wellbeing, creativity, and connection - aspects of life that have often been secondary to economic productivity.
</Callout>


Just as AI can boost our ability to do good things, it can also increase the potential to cause harm. The same capabilities that enable scientific advancement and economic growth also create unprecedented risks in the wrong hands.

Here we’ll look at three of the largest sources of potential harm:
<Embed url="https://bd43ea21c52490dc8f76e494b4edce7e.cdn.bubble.io/f1742470234207x796355670463988200/3-risks.svg"/>
<Callout title="💡 Bonus: What about other harms?">
Current AI systems are already causing harm through malfunctions—for example, making [biased](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) or [incorrect decisions](https://www.theguardian.com/technology/2023/jun/23/two-us-lawyers-fined-submitting-fake-court-citations-chatgpt).

Additionally, training and using AI systems can have negative effects on the environment ([although these are smaller than most people think](https://andymasley.substack.com/p/individual-ai-use-is-not-bad-for)). Collecting data to train AI systems can involve [exposing low-paid data annotators to graphic content](https://theconversation.com/ai-is-a-multi-billion-dollar-industry-its-underpinned-by-an-invisible-and-exploited-workforce-240568) or [using copyrighted content with unclear legal status](https://www.bakerlaw.com/services/artificial-intelligence-ai/case-tracker-artificial-intelligence-copyrights-and-class-actions/).

We think these are serious problems for current AI systems. However, we focus on catastrophic AGI risks below due to their:

- **Scale and irreversibility**: While current harms affect individuals and communities, AGI risks could impact all of humanity and might be impossible to reverse. Once we lose control to AI systems or establish AI-enabled authoritarian regimes, there may be no path back.
- **Neglectedness**: [Far more people are working on current harms from AI](https://duco-public-static-assets.s3.amazonaws.com/Duco+TnS+MRR-+FINAL.pdf), and [many partial solutions exist](https://blogs.law.ox.ac.uk/oblb/blog-post/2024/07/addressing-ai-related-harms-through-existing-tort-doctrines). In contrast, far fewer people are working on catastrophic AGI risks, and there has been much less progress in addressing them despite their potential severity.
</Callout>



When billions have access to powerful AI capabilities, even if only 0.0001% misuse it, that's thousands of bad actors with unprecedented power.

Malicious actors might be able to use AI's capabilities to:
- Conduct large-scale cyberattacks
- Design novel chemical weapons
- Coordinate sophisticated terror attacks

Today's AI models already help terrorists and hostile states generate revenue. Deepfakes have enabled scams where criminals use someone's social media videos to clone their voice. They can then contact people's relatives or colleagues, faking an emergency to get money.

For example, [North Korea raises half the foreign currency for its nuclear weapons through cyber theft](https://asia.nikkei.com/Spotlight/N-Korea-at-crossroads/North-Korea-gets-half-its-foreign-currency-from-cyber-theft-U.S.-official). In 2024, they [started using AI to conduct these attacks](https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/).

<Callout title="💡 Bonus: $25 million stolen after a deepfake video call">
In 2024, a major engineering firm [lost $25 million](https://www.ft.com/content/b977e8d4-664c-4ae4-8a8e-eb93bdf785ea) after fraudsters used deepfake technology to impersonate the company's chief financial officer and other senior executives during a video call. Fraudsters managed to convince a member of the finance team to send payments to several different bank accounts, and the police have not arrested the culprits or recovered the money.
</Callout>
<Callout title="💡 Bonus: The first physical terror attack planned by AI?">

In January 2025, an on-leave soldier [used AI to plan an attack](https://apnews.com/article/tesla-cybertruck-explosion-trump-hotel-las-vegas-248b41d87287170aa7b68d27581fdb4d) on the Trump International Hotel in Las Vegas. He loaded a Tesla Cybertruck with explosives that injured seven bystanders.

A Las Vegas sheriff investigating the case [noted](https://futurism.com/the-byte/cybertruck-bomber-chatgpt#:~:text=%22This%20is%20the%20first%20incident%20that%20I%E2%80%99m%20aware%20of%20on%20US%20soil%20where%20ChatGPT%20is%20utilized%20to%20help%20an%20individual%20build%20a%20particular%20device%2C%22%20said%20Las%20Vegas%20Metropolitan%20Police%20Department%20sheriff%20Kevin%20McMahill%20during%20a%20press%20conference.%20%22It%E2%80%99s%20a%20concerning%20moment.%22): "I think this is the first incident on US soil where ChatGPT is utilised to help an individual build a particular device."

<Embed url="https://www.youtube.com/embed/h4Q\_o7wuf3I"/>

Given the circumstances of the attack, it seems unlikely that AI assisted this individual more than a standard Google search would have done. However, it does demonstrate that people do turn to AI assistants to help cause harm – and it is concerning when combined with the potential for more harmful capabilities in the future.
</Callout>
<Callout title="💡 Bonus: Designing 40,000 chemical weapon candidates in a few hours">
In 2022, researchers flipped a drug discovery AI to optimise for harm rather than safety. Within six hours, it generated designs for [40,000 potentially lethal compounds](https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx), including some similar to VX nerve agent and other known chemical weapons.

Today's AI could help someone [make these chemicals](https://futureoflife.org/wp-content/uploads/2024/02/FLI_AI_and_Chemical_Bio_Weapons.pdf). AGI could go further—potentially synthesising these chemicals via [cloud laboratories](https://archive.ph/oUyJV) or robotics.
</Callout>

However, current AI systems are still limited in many important domains—such as planning projects end-to-end or reliably using computers. Many of the above attacks have therefore still required skilled humans to execute successfully.
Future AGI systems will be more capable. If bad actors have access, this will likely:
- Reduce the barriers to entry, so a wider range of people can conduct attacks
- Increase the capability ceiling, so these attacks will be more significant

### Good people with AGI vs bad people with AGI doesn’t end well
A common thought on AGI misuse is: “Why don’t we share AGI with everyone?” If good people have AGI too, won't they stop the bad actors?

It’s a good intuition, and is likely true for some harms. For example, in cybersecurity, it seems likely that companies could use powerful AI systems to fix their problems before cyberattackers exploit them.

However, some threats may be much harder to defend than they are to create. An example might be nuclear weapons: if every person had access to nukes, things would not go well—even if most of them were good people.

Humans are also vulnerable in ways that are hard to defend against—while you can fix security problems with a computer system, it’s hard to make all humans invulnerable against chemical or biological weapons.

This dynamic creates trade-offs between security and access. One possible approach may be to focus on sharing what AGI creates rather than the AGI itself. For instance, sharing medical treatments discovered by AI while maintaining careful control of the AI that produced them.


AGI could give states incredible military, intelligence, and political strength. Just as some states hope to use AI to crush terrorists and enforce law and order, states could use AI to crush political opponents and enforce brutal regimes.

Authoritarian governments could use AI to control their populations without the need to enlist millions of citizens to do this work. Many previous dictatorships have been toppled in moments of vulnerability like the death of a dictator, but AIs—which would be hard to “kill”—could prevent opportunities for reform.

In past authoritarian regimes, governments have had to look after their citizens somewhat—otherwise, their country would have no economic or military power. But with humans made obsolete by AGI in these roles, countries might have no incentive to look after their populations.

We have some models for what happens when a majority of the population is unimportant for a country’s economic success—states dependent on oil often succumb to a ‘resource curse’ where they have higher inequality, lower rates of democracy and poorer development outcomes.

We’re not just talking about states that traditionally have had low levels of democracy here. AGI could result in such rapid power shifts that it could lead to authoritarian regimes in traditionally ‘stable’ countries.

<Callout title="💡 Bonus: Governments might also be incentivised to go to war">
<Embed url="https://www.youtube.com/embed/w9npWiTOHX0"/>

International relations scholars often find "[large, rapid shifts in the distribution of power lead to war](https://gnss.mcgill.ca/pages/powell.pdf)." When faced with a potential power shift, states may need to choose between (i) declaring war while a rival is relatively weak, or (ii) later being threatened by a more powerful rival. AGI's ability to accelerate economic and technological development could lead to one of these power shifts.

Separately, AI could escalate conflicts. Decision-makers might [trust flawed recommendations about escalation](https://www.rand.org/pubs/perspectives/PE296.html), lethal autonomous weapons could inadvertently [initiate or expand violent conflicts](https://www.rand.org/pubs/research_reports/RR2797.html), and faster AI decisions might [encourage escalation](https://hai-production.s3.amazonaws.com/files/2024-05/Escalation-Risks-LLMs-Military-Diplomatic-Contexts.pdf) and reduce time available to de-escalate situations.
</Callout>
<Exercise exerciseId="receHnrWAIgji21HG" />


Not only do we have to prevent misuse by actors small and large, but we’ll need to make sure humans remain in control of AGI—rather than the other way around. Just as a state could use AGI to dominate populations, an AGI could likely dominate humanity if it wanted to.

Researchers often find that AI systems aren’t doing what we wanted them to. 

For example, OpenAI trained a simulated robot hand to grasp a ball, with humans evaluating success via a camera feed. Instead of actually grasping the ball, the system learned to position its hand in front of the ball—tricking humans into thinking the AI was successful.

<Embed url="https://www.youtube.com/embed/FAXMVqtsUqU?si=ZK7G1E0WvabhAdvp"/>

And with more capable systems everywhere, tasks become harder for humans to verify—or humans become complacent. (Did you really understand the code the AI model produced in Unit 1?) This, combined with AI's growing ability to deceive or persuade people, could make it hard to detect when our systems are misbehaving.

This becomes a much bigger problem when AI systems behave more like **agents** than tools. Agents pursue high-level goals more autonomously and take actions in the world—you tell them what to achieve, rather than what to do.

Additionally, AI systems have incentives to pursue resources and power almost regardless of their primary goal. In simple terms: if you want to get things done, having more money, more power, and not being stopped almost always helps.

### Case study: Modern AI power-seeking

A [recent study](https://www.apolloresearch.ai/research/scheming-reasoning-evaluations) set up an environment where AI systems were tasked with helping a company pursue clean energy. Researchers then gave the AI access to emails which included a mention of humans replacing it with a new model that pursues fossil fuels.

In this situation, the AI systems sometimes:

- Attempted to disable oversight mechanisms  
- Copied themselves to different systems  
- Lied to human operators  

This might seem like sci-fi, but it's a genuine scientific concern that even the AI companies recognise:

> "Our models may learn misaligned behaviours such as power-seeking, sandbagging, deception, and strategic scheming. […] it will be hard to measure in practice when models begin hiding their intent, and so even if [some of the latest safety techniques are] used, we recommend treading with extreme caution."
— [OpenAI research paper](https://openai.com/index/chain-of-thought-monitoring/) (March 2025)

> "The alarm bell I'm ringing has to do with the existential threat of them taking control. I used to think it was a long way off, but I now think it's serious and fairly close. 
> I think [AGI will] very quickly realise that getting more control is a very good subgoal because it helps you achieve other goals. And if these things get carried away with getting more control, we're in trouble.
> And if [AI models] are much smarter than us, they'll be very good at manipulating us. You won't realise what's going on. So even if they can't directly pull levers, they can certainly get us to pull levers. It turns out if you can manipulate people, you can invade a building in Washington without ever going there yourself."
— [Geoffrey Hinton](https://www.forbes.com/sites/craigsmith/2023/05/04/geoff-hinton-ais-most-famous-researcher-warns-of-existential-threat/), one of the "Godfathers of AI", on stepping down from Google


AGI could dramatically accelerate progress in almost all domains, lifting millions out of poverty, eradicating diseases, developing plentiful renewable energy and much more.

However, we can’t achieve these benefits if the world is in ruins. And AGI threatens to put us there by enabling:

- **Terrorism**: bad actors using AI to conduct cyberattacks or build chemical weapons  
- **Authoritarianism**: states lose the incentive to look after their populations and slide into dictatorship  
- **Loss of control**: we lose control to AI systems that are taking actions against our interests  

In the next unit we’ll look at ways to address these risks so that we can unlock the benefits.


Future of AI

Loss of Control