Technical trends driving AI progress

Intelligence Explosion

Will AI progress accelerate even faster?

Map the threat landscape

Prioritising threat pathways

Option 1: Power concentration

Option 2: Gradual disempowerment

Option 3: Catastrophic pandemics

Option 4: Critical infrastructure collapse

Imagining a better future

What future do you want?

Steering the race to AGI

The characters

What might success look like?

Building defences

Layer 1: Prevent dangerous AI training

Layer 2: Constrain dangerous AI capabilities

Layer 3: Withstand dangerous AI actions

Choose your focus

Go deep in one area

Make a plan

Create your 1-pager (optional)

Optional advice

Next steps: Deeper-dive courses

Certificate

What should we do if superhuman intelligence becomes "too cheap to meter"?

In this unit, we examine why AI capabilities keep getting better: cheaper and more plentiful compute, and better algorithms. We then analyse what that implies for the rate of future AI capabilities, and how quickly those capabilities will spread to many actors all over the world.

**Definitions:**
- [FLOP](https://blog.heim.xyz/flop-for-quantity-flop-s-for-performance/): a single basic mathematical operation (like addition or multiplication) that a computer performs. The total number of FLOP is a measure of computational work done during AI training.
- [Compute efficiency](https://www.governance.ai/research-paper/increase-compute-efficiency-and-the-diffusion-of-ai-capabilities) is a measure of how “good” an AI model you get with a given financial investment. 
    - E.g. if OpenAI spends $500M on compute to train their next AI model, how "good" will it be?
- [Hardware price-performance](https://epoch.ai/blog/trends-in-machine-learning-hardware) measures the amount of computational resources (FLOP) available for a given financial investment. 
    - E.g. how many FLOP would OpenAI get with a $500M investment into [NVIDIA H200 AI chips](https://www.nvidia.com/en-gb/data-center/h200/) running for 1 month?
    - This is improving at ~1.4x/year (i.e. you get 1.4x more FLOP per $ each year) due to Moore's Law.
- [Algorithmic efficiency](https://epoch.ai/blog/algorithmic-progress-in-language-models) measures how “good” of an AI model you get with a given number of FLOP. 
    - E.g. if OpenAI run their [NVIDIA H200 chips](https://www.nvidia.com/en-gb/data-center/h200/) for 1 month training an AI model (~10^26 FLOP), how “good” will that AI model be?
    - This is improving at ~3x/year (i.e. you need 3x less FLOP to achieve the same capability each year).
- **Compute efficiency** (Capability/$) = **Hardware price-performance** (FLOP/$) × **Algorithmic efficiency** (Capability/FLOP)
    - This is improving at ~4x/year (i.e. you need 4x less $ to achieve the same capability each year)


> Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion.'

— [Irving J Good, 1965](https://www.historyofinformation.com/detail.php?id=2142)


**Test your understanding of the content**. Don’t use jargon or fancy words — provide your explanation in simple english.

Start by defining these terms from memory: 
- FLOP
- Compute efficiency
- Hardware price-performance
- Algorithmic efficiency

Before continuing, go back to the materials and compare your response. 


Why are AI capabilities advancing at such a rapid rate?

First, what are the trends driving rapid AI progress?

Second, what are the economic, social, technological and geopolitical implications of rapid AI progress? What impact might this technology have on the world?


AGI Strategy

Intelligence Explosion

Resources (1 hr 10 mins)