Why does AGI matter?

What benefits could AGI bring?

What risks does AGI pose?

Terrorism

Authoritarianism

Loss of Control

Summary

Three ways you can contribute

1. Use your career to reduce AI risk 

2. Advocate for sensible AI regulation 

3. Donate to impactful AI safety organizations 

Your AI Safety Action Plan

What can the most powerful AI systems do today?

The speed of progress is unprecedented

What does agency look like?

So... how do AI systems work?

What is AGI?

The Vending-Bench evaluation

How could AGI get built?

AI Agents at Work

Teaching AI the next action

You might be living through one of the most important technological transitions in human history.

The development of AGI could transform society in profound ways.

Throughout history, major technologies have changed not just what humans could do, but how we live:
- 🔥 **Fire** gave us warmth, cooking, and tools—but also enabled violence.
- 🌾 **Agriculture** led to cities and stability—but also health challenges and social divisions.
- 🛢️ **Fossil fuels** powered industrial growth—while contributing to a climate crisis.
- ☢️ **Nuclear technology** brought new energy and medicine—but also weapons of mass destruction.

Each of these technologies reshaped civilization.

But advanced AI represents something **new**.

Intelligence—our ability to solve problems, create, and collaborate—is what sets humans apart from other species.

Now, for the first time, we’re building systems that may match or surpass **human thinking** across many domains.

AGI might not arrive tomorrow—or in the exact form people imagine.

But its potential benefits and risks are so large that **understanding it matters to everyone**.


AGI could **supercharge human progress**, especially in science, medicine, and economic growth.

Let’s explore how.
### 🧪 Transforming scientific discovery
Today, science is limited by human time and attention.

Even brilliant researchers can only read so many papers, analyze so much data, and work so many hours.

AGI could change this by creating **millions of virtual researchers**, working 10 to 100 times faster than humans.

These systems wouldn’t just analyze—they could:
- Design experiments
- Connect ideas across disciplines
- Make new discoveries at scale

Many breakthroughs—like CRISPR, mRNA vaccines, and brain-computer interfaces—came from **unexpected connections**.

AGI could find these patterns systematically, and at global scale.

Imagine if we could:
- Cure all cancers in days instead of decades
- Deliver personalized medicine within minutes
- Stabilize global temperatures in months
- Achieve fusion energy within a year
- Reverse aging in our lifetime
- Solve whatever problem you care most about

Some researchers call this the **“compressed 21st century” effect**: 50 to 100 years of scientific progress packed into just 5 to 10 years.

### ⚗️ Real examples of AI scientific progress
Even before AGI, we’ve seen major AI-driven breakthroughs:
- **AlphaFold** solved the protein-folding problem, accelerating biology.
- Google’s AI diagnostic tool **outperforms doctors** in breast cancer detection.
- AI systems discovered new antibiotics like **Halicin** and **Abaucin**.
- **AlphaTensor** improved how computers do matrix math.
- **FourCastNet** predicts weather **weeks ahead** with unprecedented speed and accuracy.

But even with AGI, **some things still take time**:
- Physical experiments
- Clinical trials
- Real-world adoption

AGI may help speed these up—but **can’t skip them entirely**.
### 
### 💰 Economic growth and prosperity
AI is already boosting the productivity of millions of workers:
- Companies using AI report **8–30% gains** in productivity
- AI customer support agents handle **14% more issues per hour**
- Google cut **data center cooling energy** by 40% using AI
- AI could add **$4.4 trillion** to the world economy each year

AGI could go even further by enabling:
- Massive productivity gains across all sectors
- Discovery of new materials, energy sources, and manufacturing methods
- Optimization of transport, supply chains, and resource use

Some economists suggest AGI could lead to **10–20% growth per year**.

For perspective:
- At 2% growth, living standards double every **35 years**
- At 15% growth, they double every **5 years**

This isn’t just about faster growth—it’s a shift from **scarcity** to **abundance**.

Goods and services once available only to the wealthy could become **universally accessible**.

Think about how smartphones went from a luxury item to being used by billions of people. And imagine that for every item of material wealth.

### 🏛️ Better functioning societies
AGI might also help our societies run better by:
- Detecting corruption and inefficiencies
- Supporting informed democratic decision-making
- Delivering public services tailored to individual needs

Used well, AGI could lead to **more just, efficient, and responsive governance**.

<Callout title="💡 Bonus: What does the future of human labour look like?">
AI won't just automate routine tasks; it will increasingly perform work once thought uniquely human—creative, analytical, and interpersonal. Within a single generation, we may face a profound change in how our economy works.

As AGI develops, even careers we consider secure—doctors, lawyers, artists, scientists—could face disruption, alongside manual trades. This shift might lead not just to temporary unemployment but to a lasting disconnect between human skills and economic value for many people.

The benefits of AGI might flow disproportionately to those who own and control these technologies, potentially creating unprecedented concentrations of wealth. This economic power often translates to political influence, which could further widen these gaps.

Throughout history, our economic systems have distributed resources largely based on people's contributions through work. What happens in a world where many people's labour might no longer have significant market value? This raises profound questions about how society should function.

This possible future challenges us to rethink not only economics but also human identity and purpose. We may need new economic models, thoughtful policies for more equitable distribution of AI's benefits, and perhaps a fundamental shift in how we value human contributions beyond paid work.

While these challenges are serious, they also create an opportunity to build a society that places greater emphasis on human wellbeing, creativity, and connection—aspects of life that have often been secondary to economic productivity.
</Callout>


Think of a specific challenge in your life or a problem you care about (healthcare costs, climate change, education access). How might advanced AI help solve this particular issue?


Personal Impact

If AI creates tremendous new wealth and opportunities, how do you think this wealth and access to these opportunities will be distributed by default? How do you think they should be distributed?


Equal Access

Just as AI can boost our ability to do good things, it can also increase the potential to cause harm. The same capabilities that enable scientific advancement and economic growth also create unprecedented risks in the wrong hands.
Here we’ll look at three of the largest sources of potential harm:
<Embed url="https://bd43ea21c52490dc8f76e494b4edce7e.cdn.bubble.io/f1742470234207x796355670463988200/3-risks.svg"/>
<Callout title="💡 Bonus: What about other harms?">
Current AI systems are already causing harm through malfunctions—for example, making [biased](https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing) or [incorrect decisions](https://www.theguardian.com/technology/2023/jun/23/two-us-lawyers-fined-submitting-fake-court-citations-chatgpt).

Additionally, training and using AI systems can have negative effects on the environment ([although these are smaller than most people think](https://andymasley.substack.com/p/individual-ai-use-is-not-bad-for)). Collecting data to train AI systems can involve [exposing low-paid data annotators to graphic content](https://theconversation.com/ai-is-a-multi-billion-dollar-industry-its-underpinned-by-an-invisible-and-exploited-workforce-240568) or [using copyrighted content with unclear legal status](https://www.bakerlaw.com/services/artificial-intelligence-ai/case-tracker-artificial-intelligence-copyrights-and-class-actions/).

We think these are serious problems for current AI systems. However, we focus on catastrophic AGI risks below due to their:

- **Scale and irreversibility**: While current harms affect individuals and communities, AGI risks could impact all of humanity and might be impossible to reverse. Once we lose control to AI systems or establish AI-enabled authoritarian regimes, there may be no path back.
- **Neglectedness**: [Far more people are working on current harms from AI](https://duco-public-static-assets.s3.amazonaws.com/Duco+TnS+MRR-+FINAL.pdf), and [many partial solutions exist](https://blogs.law.ox.ac.uk/oblb/blog-post/2024/07/addressing-ai-related-harms-through-existing-tort-doctrines). In contrast, far fewer people are working on catastrophic AGI risks, and there has been much less progress in addressing them despite their potential severity.
</Callout>



When billions have access to powerful AI capabilities, even if only 0.0001% misuse it, that's thousands of bad actors with unprecedented power.

Malicious actors might be able to use AI's capabilities to:
- Conduct large-scale cyberattacks
- Design novel chemical weapons
- Coordinate sophisticated terror attacks

Today's AI models already help terrorists and hostile states generate revenue. Deepfakes have enabled scams where criminals use someone's social media videos to clone their voice. They can then contact people's relatives or colleagues, faking an emergency to get money.
For example, [North Korea raises half the foreign currency for its nuclear weapons through cyber theft](https://asia.nikkei.com/Spotlight/N-Korea-at-crossroads/North-Korea-gets-half-its-foreign-currency-from-cyber-theft-U.S.-official). In 2024, they [started using AI to conduct these attacks](https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/).
<Callout title="💡 Bonus: $25 million stolen after a deepfake video call">
In 2024, a major engineering firm [lost $25 million](https://www.ft.com/content/b977e8d4-664c-4ae4-8a8e-eb93bdf785ea) after fraudsters used deepfake technology to impersonate the company's chief financial officer and other senior executives during a video call. Fraudsters managed to convince a member of the finance team to send payments to several different bank accounts, and the police have not arrested the culprits or recovered the money.
</Callout>
<Callout title="💡 Bonus: The first physical terror attack planned by AI?">
In January 2025, an on-leave soldier [used AI to plan an attack](https://apnews.com/article/tesla-cybertruck-explosion-trump-hotel-las-vegas-248b41d87287170aa7b68d27581fdb4d) on the Trump International Hotel in Las Vegas. He loaded a Tesla Cybertruck with explosives that injured seven bystanders.

A Las Vegas sheriff investigating the case [noted](https://futurism.com/the-byte/cybertruck-bomber-chatgpt#:~:text=%22This%20is%20the%20first%20incident%20that%20I%E2%80%99m%20aware%20of%20on%20US%20soil%20where%20ChatGPT%20is%20utilized%20to%20help%20an%20individual%20build%20a%20particular%20device%2C%22%20said%20Las%20Vegas%20Metropolitan%20Police%20Department%20sheriff%20Kevin%20McMahill%20during%20a%20press%20conference.%20%22It%E2%80%99s%20a%20concerning%20moment.%22): "I think this is the first incident on US soil where ChatGPT is utilised to help an individual build a particular device."

<Embed url="https://www.youtube.com/embed/h4Q\_o7wuf3I"/>

Given the circumstances of the attack, it seems unlikely that AI assisted this individual more than a standard Google search would have done. However, it does demonstrate that people do turn to AI assistants to help cause harm – and it is concerning when combined with the potential for more harmful capabilities in the future.
</Callout>
<Callout title="💡 Bonus: Designing 40,000 chemical weapon candidates in a few hours">
In 2022, researchers flipped a drug discovery AI to optimise for harm rather than safety. Within six hours, it generated designs for [40,000 potentially lethal compounds](https://www.theverge.com/2022/3/17/22983197/ai-new-possible-chemical-weapons-generative-models-vx), including some similar to VX nerve agent and other known chemical weapons.

Today's AI could help someone [make these chemicals](https://futureoflife.org/wp-content/uploads/2024/02/FLI_AI_and_Chemical_Bio_Weapons.pdf). AGI could go further—potentially synthesising these chemicals via [cloud laboratories](https://archive.ph/oUyJV) or robotics.
</Callout>

However, current AI systems are still limited in many important domains—such as planning projects end-to-end or reliably using computers. Many of the above attacks have therefore still required skilled humans to execute successfully.
Future AGI systems will be more capable. If bad actors have access, this will likely:
- Reduce the barriers to entry, so a wider range of people can conduct attacks
- Increase the capability ceiling, so these attacks will be more significant

### Good people with AGI vs bad people with AGI doesn’t end well
A common thought on AGI misuse is: “Why don’t we share AGI with everyone?” If good people have AGI too, won't they stop the bad actors?

It’s a good intuition, and is likely true for some harms. For example, in cybersecurity, it seems likely that companies could use powerful AI systems to fix their problems before cyberattackers exploit them.

However, some threats may be much harder to defend than they are to create. An example might be nuclear weapons: if every person had access to nukes, things would not go well—even if most of them were good people.

Humans are also vulnerable in ways that are hard to defend against—while you can fix security problems with a computer system, it’s hard to make all humans invulnerable against chemical or biological weapons.

This dynamic creates trade-offs between security and access. One possible approach may be to focus on sharing what AGI creates rather than the AGI itself. For instance, sharing medical treatments discovered by AI while maintaining careful control of the AI that produced them.


AGI could give states incredible military, intelligence, and political strength. Just as some states hope to use AI to crush terrorists and enforce law and order, states could use AI to crush political opponents and enforce brutal regimes.

Authoritarian governments could use AI to control their populations without the need to enlist millions of citizens to do this work. Many previous dictatorships have been toppled in moments of vulnerability like the death of a dictator, but AIs—which would be hard to “kill”—could prevent opportunities for reform.

In past authoritarian regimes, governments have had to look after their citizens somewhat—otherwise, their country would have no economic or military power. But with humans made obsolete by AGI in these roles, countries might have no incentive to look after their populations.

We have some models for what happens when a majority of the population is unimportant for a country’s economic success—states dependent on oil often succumb to a ‘resource curse’ where they have higher inequality, lower rates of democracy and poorer development outcomes.

We’re not just talking about states that traditionally have had low levels of democracy here. AGI could result in such rapid power shifts that it could lead to authoritarian regimes in traditionally ‘stable’ countries.

<Callout title="💡 Bonus: Governments might also be incentivised to go to war">
<Embed url="https://www.youtube.com/embed/w9npWiTOHX0"/>

International relations scholars often find "[large, rapid shifts in the distribution of power lead to war](https://gnss.mcgill.ca/pages/powell.pdf)." When faced with a potential power shift, states may need to choose between (i) declaring war while a rival is relatively weak, or (ii) later being threatened by a more powerful rival. AGI's ability to accelerate economic and technological development could lead to one of these power shifts.

Separately, AI could escalate conflicts. Decision-makers might [trust flawed recommendations about escalation](https://www.rand.org/pubs/perspectives/PE296.html), lethal autonomous weapons could inadvertently [initiate or expand violent conflicts](https://www.rand.org/pubs/research_reports/RR2797.html), and faster AI decisions might [encourage escalation](https://hai-production.s3.amazonaws.com/files/2024-05/Escalation-Risks-LLMs-Military-Diplomatic-Contexts.pdf) and reduce time available to de-escalate situations.
</Callout>
<Exercise exerciseId="receHnrWAIgji21HG" />


Not only do we have to prevent misuse by actors small and large, but we’ll need to make sure humans remain in control of AGI—rather than the other way around. Just as a state could use AGI to dominate populations, an AGI could likely dominate humanity if it wanted to.

Researchers often find that AI systems aren’t doing what we wanted them to. 

For example, OpenAI trained a simulated robot hand to grasp a ball, with humans evaluating success via a camera feed. Instead of actually grasping the ball, the system learned to position its hand in front of the ball—tricking humans into thinking the AI was successful.

<Embed url="https://www.youtube.com/embed/FAXMVqtsUqU?si=ZK7G1E0WvabhAdvp"/>

And with more capable systems everywhere, tasks become harder for humans to verify—or humans become complacent. (Did you really understand the code the AI model produced in Unit 1?) This, combined with AI's growing ability to deceive or persuade people, could make it hard to detect when our systems are misbehaving.

This becomes a much bigger problem when AI systems behave more like **agents** than tools. Agents pursue high-level goals more autonomously and take actions in the world—you tell them what to achieve, rather than what to do.

Additionally, AI systems have incentives to pursue resources and power almost regardless of their primary goal. In simple terms: if you want to get things done, having more money, more power, and not being stopped almost always helps.

### Case study: Modern AI power-seeking

A [recent study](https://static1.squarespace.com/static/6593e7097565990e65c886fd/t/6751eb240ed3821a0161b45b/1733421863119/in_context_scheming_reasoning_paper.pdf) set up an environment where AI systems were tasked with helping a company pursue clean energy. Researchers then gave the AI access to emails which included a mention of humans replacing it with a new model that pursues fossil fuels.

In this situation, the AI systems sometimes:

- Attempted to disable oversight mechanisms  
- Copied themselves to different systems  
- Lied to human operators  

This might seem like sci-fi, but it's a genuine scientific concern that even the AI companies recognise:

> "Our models may learn misaligned behaviours such as power-seeking, sandbagging, deception, and strategic scheming. […] it will be hard to measure in practice when models begin hiding their intent, and so even if [some of the latest safety techniques are] used, we recommend treading with extreme caution."
— [OpenAI research paper](https://openai.com/index/chain-of-thought-monitoring/) (March 2025)

> "The alarm bell I'm ringing has to do with the existential threat of them taking control. I used to think it was a long way off, but I now think it's serious and fairly close. 
> I think [AGI will] very quickly realise that getting more control is a very good subgoal because it helps you achieve other goals. And if these things get carried away with getting more control, we're in trouble.
> And if [AI models] are much smarter than us, they'll be very good at manipulating us. You won't realise what's going on. So even if they can't directly pull levers, they can certainly get us to pull levers. It turns out if you can manipulate people, you can invade a building in Washington without ever going there yourself."
— [Geoffrey Hinton](https://www.forbes.com/sites/craigsmith/2023/05/04/geoff-hinton-ais-most-famous-researcher-warns-of-existential-threat/), one of the "Godfathers of AI", on stepping down from Google


AGI could dramatically accelerate progress in almost all domains—lifting millions out of poverty, eradicating diseases, developing plentiful renewable energy and much more.

However, we can’t achieve these benefits if the world is in ruins. And AGI threatens to put us there—with:

- **Terrorism**: bad actors using AI to conduct cyberattacks or build chemical weapons  
- **Authoritarianism**: states lose the incentive to look after their populations and slide into dictatorship  
- **Loss of control**: we lose control to AI systems that are taking actions against our interests  

In the next unit we’ll look at ways to address these risks so that we can unlock the benefits.


Future of AI

Grapple with how AGI could cure diseases and automate boring work, while also posing severe risks.

How could AGI change our lives?

Learn what the most advanced AI systems can do today, and how they work.

What can AI do today?

Learn what AGI means, and why it might arrive within the next decade.

Explore the challenge of building safe AGI amid heated global competition.

Future of AI

What benefits could AGI bring?

Exercises