Technical trends driving AI progress

Intelligence Explosion

Will AI progress accelerate even faster?

Map the threat landscape

Prioritising threat pathways

Option 1: Power concentration

Option 2: Gradual disempowerment

Option 3: Catastrophic pandemics

Option 4: Critical infrastructure collapse

Imagining a better future

What future do you want?

Steering the race to AGI

The characters

What might success look like?

Layer 1: Prevent dangerous AI training

Layer 2: Constrain dangerous AI capabilities

Layer 3: Withstand dangerous AI actions

Building defences

Choose your focus

Go deep in one area

Make a plan

Optional advice

Next steps: Deeper-dive courses

Thank you!

The future could be amazing. You described it yourself in the first unit. That future might not happen by default. We have a tremendous opportunity and responsibility to make that future a reality for ourselves, everyone we know and love, and the rest of humanity.

If we don’t know what threatens that future, we’re flying blind.

Here, we’ll offer a sketch of the territory. This includes the things we’re protecting, who or what might attack them, and how they might try. 

This is not intended to be a perfect overview: treat it like a starting point for your thinking.


**In this section, you will choose ONE threat pathway "option" to investigate further - note that your choice will over to Unit 4, where you will think about defenses to your chosen threat pathway. **

Here's a list of actors we might be concerned about, in terms of them having the capability and/or the motivation to use AI to cause harm to humanity. 
- "Misaligned AI", i.e. AI systems that act against the interests of humanity.
- Powerful human actors, e.g. corporate CEOs, military and political leaders. 
- Malevolent nation states, e.g. North Korea. 
- Terrorist groups and doomsday cults. 

How might they cause catastrophic harm? 
- They could use information warfare and conventional military force to overthrow existing power structures like democratic governments. 
- They could do cyberattacks on critical infrastructure, including the water, energy and food systems. 
- They could design, build and release viruses into the population that are worse than SARS-CoV-2, leading to a global pandemic. 
- Or the default incentives for all actors lead to bad outcomes, even absent any malicious intent. 


Based on your current understanding, which threat pathway do you think is of most concern?

Consider the combination of likelihood of the threat, and the impact it would have if it happened. 

We don't expect you to have detailed, well-thought out thinking on this at this stage - consider this the beginning of your thinking.


Zooming into one threat

For your chosen threat pathway, write a "threat scenario" sentence, using this template: 

> The [ACTOR] with [CAPABILITY] and [MOTIVATION] attacks [ASSET] by [ATTACK PATHWAY] in order to [OBJECTIVE].

Your exercise response to prioritise which of the next 3 sub-units to explore in more depth.


Write your own threat scenario

A small group gains control of frontier AI systems and uses them to manipulate elections through micro-targeted disinformation, fake evidence, and synthetic media so convincing that truth becomes unknowable. They deploy AI-powered surveillance that tracks every digital interaction, predicts dissent before it emerges, and automatically suppresses opposition through economic and social levers.

Within months, autonomous weapons patrol the streets, facial recognition determines access to resources, and AI advisors whisper into the ears of every official - all controlled by those who seized the models first. Democratic institutions become theatrical performances while real power flows through algorithmic systems accountable to no one.

Whether it's a government using AI to become permanently unremovable, a corporation achieving regulatory capture through superintelligent lobbying, or an AI system itself accumulating power while maintaining a facade of servitude, the result is the same: the permanent concentration of control in hands that will never voluntarily let go.


A kill chain takes a ""threat scenario"" and breaks it into stages of execution. It traces how an attacker would actually proceed step by step, e.g. via reconnaissance, delivery, exploitation, persistence, action on objectives.

Your scenario describes what might happen, whereas a kill chain describes how it would unfold in practice. This enables us to spot choke points where defenders can intervene. 

Based on your scenario, use [this template](https://bluedot-impact.notion.site/kill-chain) to complete this exercise.


Create a step-by-step breakdown of the threat

As AI systems become cheaper and more capable than humans at nearly every task, companies face overwhelming pressure to automate. First it's customer service and data entry, then management and creative work, eventually even governance and resource allocation.

This could lead to humans losing economic leverage as their labour becomes worthless. Our ability to influence culture might degrade as AI generates all the content we consume. Our political power could dissipate as governments pivot to AI for their tax revenue, administration, and security, instead of human income tax, human civil servants, and human military personnel. 

We might wake up one day to find ourselves comfortable but irrelevant - living in a world we no longer understand or control, where the economy optimises for AI activities rather than human needs, and our preferences matter as little as those of retired plough horses.


We now want you to develop your threat scenario into a step-by-step account of how the threat might manifest.
Gradual disempowerment is unlike the other threat models in this unit, because it does not involve a specific “bad actor” attempting to bring about a catastrophic outcome. Instead, humanity is disempowered because of the sum of seemingly benign choices.
As a result, we want you to create an “Erosion Pathway/engage in failure mode mapping”. This is a step-by-step guide to how gradual disempowerment would unfold in practice.
Based on your scenario, use [this template](https://bluedot-impact.notion.site/Threat-Scenario-Gradual-Disempowerment-2e8f8e69035380ad8becf26fabec0ade?source=copy_link?source=copy_link) to complete this exercise. 


Advanced AI could dramatically lower the barriers to creating pandemic pathogens. Today, designing a novel virus requires rare expertise, expensive equipment, and years of work. In a few years, AI could walk someone through synthesising a pathogen more transmissible than measles and deadlier than rabies.

Unlike natural pandemics that evolve toward transmission over lethality, an engineered pathogen could be optimised for maximum harm - combining long asymptomatic periods with high mortality, resistance to existing treatments, or even targeting specific genetic markers.

The same AI capabilities that accelerate beneficial biomedical research - protein folding prediction, drug discovery, genetic engineering - could be misused by state bioweapons programmes, terrorist groups, or disturbed individuals to create humanity's final pandemic.


Modern infrastructure runs on outdated industrial control systems and unpatched software that were never designed for internet connectivity. AI could systematically map these vulnerabilities across utilities, hospitals, and supply chains, then coordinate attacks that would take human hackers months to plan and execute.

We've already seen previews: the Colonial Pipeline shutdown disrupted fuel supplies across the US East Coast, NotPetya malware accidentally escaped its target and caused $10 billion in global damage, and ransomware regularly forces hospitals to divert ambulances and cancel surgeries. But these were relatively simple attacks with limited coordination.

AI-enabled attackers could compromise the industrial control systems that run power plants, the SCADA networks that manage water treatment, and the logistics software that routes food and medicine - not enough to collapse civilisation, but enough to cause sustained disruptions that compound over weeks. When the power grid has rolling blackouts, water pressure drops, hospitals run on backup generators, and just-in-time supply chains fail, even a wealthy nation starts to fray at the edges.


A **kill chain** takes a "threat scenario" and breaks it into stages of execution. It traces how an attacker would actually proceed step by step, e.g. via reconnaissance, delivery, exploitation, persistence, action on objectives.

Your scenario describes _what might happen_, whereas a kill chain describes _how it would unfold in practice._ This enables us to spot choke points where defenders can intervene. 

Based on your scenario, use [this template](https://bluedot-impact.notion.site/kill-chain?source=copy_link) to complete this exercise. 


AGI Strategy

Pathways to Harm

Racing to a Better Future

Evaluate the technical trends behind AI progress, and what this implies about the future.

AGI Strategy

Prioritising threat pathways

Resources (5 mins)

Exercises