Starting your journey

The Future of AI course

Key organisations in the AI value chain

Case study: Spotlight on AISI

Being high-agency

Turbocharging operations with AI

Prep: Applying your learning

[90 mins] Exercise: Applying your learning

Certificate

Next steps: Programs

_Understanding who builds AI, who controls it, and who keeps it safe_

In the Future of AI course, we saw how AI is advancing rapidly. But behind every AI breakthrough—from text generation to autonomous agents—lies a complex industrial ecosystem.

Understanding this ecosystem helps identify **control points** where governance can be applied. Let's dive in!
## **The AI Value Chain: From Silicon to Software**
Think of AI development like a massive restaurant operation. You need:
- **Industrial kitchens** with specialized ovens (data centres with AI chips)
- **Basic ingredients** to cook with (training data)
- **Restaurant chains** that own the kitchens (cloud providers)
- **Master chefs** who create signature dishes (model developers)
- **Local restaurants** that adapt dishes for customers (application companies)

### **Layer 1: AI Chips - Kitchen Equipment 🔌**
Training frontier models requires thousands of AI chips running trillions of computations. Just like with normal computer chips, each new generation of AI chips brings even higher processing speeds.

**The Chip Makers:**
- **NVIDIA**: Dominates with 80%+ market share in designing AI chips
- **AMD**: NVIDIA's main competitor on chip design
- **TSMC**: Actually manufactures most of the world's advanced chips, including to NVIDIA's and AMD's designs
- **Google** and **Amazon**: Developing custom AI chips for their datacenters

**The control point:** NVIDIA + TSMC essentially control who can build frontier AI. US export restrictions placed on NVIDIA leverage this chokepoint to restrict China's access.
### **Layer 2: Data - The Ingredients 📊**
**Training Data Providers:**
- **Scale AI**: Human feedback and data labeling (crucial for making models helpful)
- **Various other dataset companies**: Specialized data for different domains

**A lot of data requires a lot of people:** Getting humans to rate AI responses for quality is expensive but crucial. Scale AI [has hired hundreds of thousands of workers](https://adamjones.me/blog/300k-ai-annotators/) to provide feedback that makes models like ChatGPT actually useful rather than just coherent.
### **Layer 3: Cloud Infrastructure - The Restaurant Landlords ☁️**
**The Big Three:**
- **Microsoft Azure**: OpenAI's exclusive partner, provides compute in exchange for model access and profits
- **Amazon AWS**: Largest overall cloud provider, main provider to Anthropic
- **Google Cloud**: Third in market share but strong AI-specific tools and hardware, main provider to Google DeepMind and secondary provider to Anthropic
### **Layer 4: Foundation Model Developers - The Chefs 🧑‍🍳**
**The Leaders:**
- **OpenAI**: ChatGPT models. Often considered one of the most reckless on safety.
- **Anthropic**: Claude models. Often considered more safety-focused than competitors.
- **Google DeepMind**: Gemini and Gemma models.
- **Meta**: Llama models. They release the weights for other companies to use for free. Often considered more skeptical of AI risks.

**Emerging Players:**
- **Mistral**: European challenger, mix of open and closed models
- **xAI** (Elon Musk): Grok models, trained on Twitter data
- **Cohere**: Enterprise-focused language models
- **Various Chinese companies**: DeepSeek, Alibaba (Qwen models)

**Economies of scale:** Training runs cost $100+ million each. Most value comes from serving inference (answering user queries), not the training itself. This creates winner-take-most dynamics where the best models capture most users.

<Exercise exerciseId="reccosbWslrUUGvMJ" />
### **Layer 5: Application Layer - The Local Restaurants **🍽️
**Enterprise Productivity:**
- **Microsoft**: Copilot across Bing, Microsoft Office, Windows, built on OpenAI's models
- **Google**: AI features in Google Workspace (Gmail, Docs, Sheets), built on DeepMind's models

**Developer Tools:**
- **GitHub Copilot (Microsoft)**: AI coding assistant
- **Claude Code (Anthropic)**: AI code editing agent
- **Windsurf (OpenAI)**: AI-first code editor

**Specialized Applications:**
- **Legal**: Harvey
- **Creative**: Midjourney, Adobe Firefly

**The brutal reality:** Many AI startups will fail as base models improve and get cheaper. Success requires either strong distribution (existing users), deep domain expertise, or proprietary data that's hard to replicate.
## **AI Safety Infrastructure: Who's Watching the Watchers?**
Parallel to the commercial value chain, a network of safety institutions is emerging to evaluate AI risks and inform policy.
### **Government Safety Institutes**
**🇬🇧 UK AI Security Institute (AISI)**
- **Mission**: Equip governments with scientific understanding of advanced AI risks
- **Approach**: "Startup in government" combining government authority with private sector expertise and agility
- **Focus Areas**: Misuse (cyber/bio), safeguards effectiveness, autonomy risks, criminal applications, human influence, societal resilience

**🇺🇸 US AI Safety Institute**
- **Mission**: Develop AI safety standards and evaluation methods
- **Approach**: Part of NIST, focuses on measurement science and standards
- **Coordination**: Joint testing programs with UK AISI, including model access sharing and expert talent transfers

**🇪🇺 EU AI Office**
- **Mission**: Enforce the comprehensive EU AI Act
- **Approach**: Risk-based regulation with mandatory compliance
- **Focus Areas**: High-risk AI applications (hiring, healthcare, law enforcement)
### **Independent Safety Organizations**
**METR (Model Evaluation & Threat Research)**
- **Focus**: Testing AI models for dangerous autonomous capabilities
- **Recent work**: Evaluating whether models can self-replicate or acquire resources

**Apollo Research**
- **Focus**: AI deception and manipulation capabilities
- **Approach**: Testing whether models exhibit scheming or power-seeking behaviors

**Why independent orgs matter:** Reduced conflicts of interest with commercial AI development, and can pursue research that companies might avoid. Not beholden to political lines, and sometimes can be more nimble than governments.
## **Key Takeaways**
**The AI value chain is highly concentrated** at key bottlenecks (chips, cloud, frontier models) but fragmented at the application layer.

**New governance models** like the UK's "startup in government" approach attempt to combine technical expertise with regulatory authority, potentially creating more effective intervention points in the value chain.

<Exercise exerciseId="rec9zbu8YML82G3gU" />


This question will require Googling!

Groq (_not Grok!_) is an AI company. What part of the AI value chain do they primarily innovate in?


Grokking groq

This question will require some Googling!

Anthropic recently released their Claude 4 series of models. They activated some protections because one of the models triggered some threshold. What was this threshold?


Claude 4

> i hate how well asking myself “if i had 10x the agency i have what would i do” works
> 
> — @nickcammarata, [January 7, 2025](https://x.com/nickcammarata/status/1876749765951562209)

If you had 10x agency, what could you do to secure an AI safety operations role? (Or in other words: how could you put this into practice, today?)


10x Agency

How would you explain RepliBench to your parents?


Activity Q1

Which of these is the paper’s canary string?


Activity Q2

What is pretraining, and how could the canary string prevent the data being included in this?


Activity Q3

Activity Q4

What is an open-weights model? What’s the difference between this and an open-source model?


Activity Q5

List 3 frontier open-weights models created by **3 different** AI companies, that were released in 2025


Activity Q6

Explain the request above in plain English, and develop a project plan for this study. It should list all the stakeholders involved, what they are responsible for and when. Include details about how you will keep track of project progress.

It might be hard to do this in the small space below! Feel free to do this in a Google Doc or similar and link it in the box.


Activity Q7

Use an AI model to simulate feedback from the research lead on your project plan. The more detail you provide about your research lead’s motivations and typical behaviours, the richer the feedback will be.

Briefly summarise the most useful feedback you received in the box below.


Activity Q8

[Download the log files here](https://drive.google.com/uc?export=download&id=1SJ34WiuuoSzoUGfZkDmlDxPj4WNu8q0T). Then, extract the relevant data from the log files and put it into [this Google Sheet template](https://docs.google.com/spreadsheets/d/1av9w73MZ_3ONGMgHIUdg6tofsTJzqphvP7AnFVq4DOM/copy).

The team thinks they might have to do this again for a future study, so making this process easy to do in future would be a bonus.

Link your Google Sheet, plus anything else you create, in the box below.


Activity Q9

AISI's responsibilities

<Embed url="https://www.youtube.com/embed/xqdmv2anU0o" />

The UK AI Safety Institute (AISI) is the world's largest government-backed AI safety research lab. They evaluate the most advanced AI systems to understand their capabilities and potential risks, then communicate these findings to policymakers to inform evidence-based decisions on AI governance and safety.

**Overview of the AISI workstreams**
- **Human Influence:** This team investigates how AI systems can manipulate, persuade, deceive or subtly steer human behaviours, for example by running large-scale human studies to measure these impacts.
- **Cyber and Autonomous Systems:** This team maps dangerous AI capabilities that could threaten cybersecurity and human control over autonomous systems, for example developing evaluations for whether AI systems can copy themselves across systems.
- **Societal Resilience:** This team studies how AI capabilities might be used for harm, and interventions we can put in place to protect the world - for example, collaborating with the above teams to safeguard democratic institutions or AI-proofing critical national infrastructure from cyberattacks.
- **Criminal Misuse:** This team assesses how AI systems could enable criminal activities beyond current internet capabilities, for example evaluating whether models comply with requests for fraud, embezzlement, or social engineering schemes.
- **Dual-use Science:** This team evaluates risks from AI systems highly capable at scientific tasks, focusing on potential misuse in chemical, biological, and other sensitive research domains.
- **Science of Evaluations:** This team develops rigorous scientific techniques for measuring AI capabilities, ensuring evaluations are accurate, robust, and useful for decision-making.
- **Capabilities Post-Training:** This team uses cutting-edge techniques to reveal the true performance ceiling of AI systems, ensuring we don't underestimate frontier capabilities.
- **Safeguard Analysis:** This team stress-tests AI safety measures through adversarial attacks and jailbreaking, identifying vulnerabilities and developing best practices for robust defences.
- **Control:** This team researches methods to maintain human control over AI systems deployed in high-stakes environments, developing protocols to prevent autonomous harmful actions.
- **Alignment:** This team works on ensuring AI systems remain honest and aligned with human values as they scale beyond human-level intelligence.

_([See here](https://www.aisi.gov.uk/research-agenda) for more details)_

<Exercise exerciseId="recYposz4lyY5elPH" />



In the previous section, we explored the extraordinary importance of AI safety organisations. To succeed in their mission despite the many challenges they face, they’ll need people who take extraordinarily resourceful - what's often called being "high-agency."

High-agency people often ([from Andrew Yeung](https://www.andrew.today/p/what-it-means-to-be-high-agency)):
- **Challenge the status quo.** They challenge social norms and beliefs and regularly ask, “Why?". They have strong convictions on important topics, think independently of the people around them, and take action to follow their curiosity. Examples may include leaving a traditional corporate job, “thinking outside the box”, and finding unconventional methods to accomplish things.
- **Are _treadmill_ people rather than _couch people_.** Treadmill people give you energy. You feel incredible after spending time with them—like you can do anything in the world. The opposite of that is couch people. You might feel exhausted after spending time with them and want to lay on the couch for the rest of the day.
- **Confound stereotypes.** The investment banker who writes French poetry. The software engineer who paints in watercolor. The gymnast who moonlights as a data scientist. If their actions and beliefs don’t align with their stereotypes, then it’s likely they’ve demonstrated agency (and the ability to think and act independently of their surroundings.)
- **Lack limiting beliefs.** _“I can’t do it”_ isn’t in their vocabulary. Nothing is too hard or too complex. They always believe that they will find a way to do it, and most of the time, they’re right. They are _radically optimistic_ people.
- **The prison question.** If you had to call someone to break you out of a prison in a far-flung country, who would you call? This is probably the highest agency person you know. (from [George Mack](https://www.highagency.com/))

Being high-agency isn't an inherent trait you're born with, but rather a mindset to problem-solving that can be strengthened over time. You can escape the common low-agency traps (vagueness, overthinking, and attachment to past assumptions) through specific practices: defining problems clearly outside your head, taking the smallest possible first step, questioning your assumptions, focusing on actions within your control. The more you practice making things happen, the stronger your high-agency muscle becomes.

<Exercise exerciseId="recZ1vTSue0s04gbg" />

This developmental mindset is essential in rapidly evolving fields like AI safety, where creative problem-solving under constraints is the norm. The more tools and skills you have at your disposal, the more creative solutions you can devise.

This is precisely why AI tools are becoming such powerful enablers of high-agency behavior. In the next section, we'll explore how these tools can dramatically expand your capabilities, helping you become more productive, resourceful, and effective in tackling complex operational challenges.


This exercise will explore RepliBench, some recent work by AISI’s cyber and autonomous systems team. 

To begin, [read AISI’s blog article on RepliBench](https://www.aisi.gov.uk/work/replibench-measuring-autonomous-replication-capabilities-in-ai-systems).

<Exercise exerciseId="recoqey4UHOg0bosS" />

The paper and dataset include canary strings to prevent the benchmark from being included in pretraining data.

<Exercise exerciseId="rec3A4DDpIXM91NBR" />

<Exercise exerciseId="recd9VHAOWJSSZx8F" />

<Exercise exerciseId="recDI6cQr9WdNgCVM" />

The RepliBench study focused on models behind APIs, such as Anthropic’s Claude 3.7 Sonnet and OpenAI’s o3-mini.

The researchers want to commission a follow-up study with open-weights models.

<Exercise exerciseId="recufh8lAEYJVEtIL" />

<Exercise exerciseId="recit6ar9fkaQiaoz" />

You receive the following message from the research lead of the autonomous systems team:
> I'd like to conduct a follow-up study using top open-weights models to evaluate their autonomous replication capabilities. The timeline is tight - I'd like to complete this within 8 weeks if possible.
> 
> I'll need about five H100s for 3-4 weeks to run all the evaluations. We should hire 2 RE contractors with experience in model scaffolding and deployment to implement the necessary agent frameworks for the models.
> 
> I'll need legal to review the licensing for all these models to ensure we can do this and publish results.

<Exercise exerciseId="recfLcVMLQlYI0c1A" />

<Exercise exerciseId="rec0AEGlY0UASshsq" />

Great job! Your research lead has approved your plan and the project is underway. A research engineer (and definitely not just Adam in a funny outfit) asks for an urgent call about work needed to hit the publication deadline...

<Embed url="https://www.youtube.com/embed/dBOPzd35w7U" />

<Exercise exerciseId="recqkKwlCH144opOh" />


We'll email learners on the facilitated course a certificate after the second discussion! You'll then be able to access your certificate here.


AI Safety Operations Bootcamp

Turbocharging operations with AI