Resources: Mechanistic interpretability

Exercises: Mechanistic interpretability

Resources: Rapidly testing your project

Exercises: Rapidly testing your project

Resources: Building in public

Exercises: Building in public

Resources: Developing your project

Exercises: Developing your project

Resources: Technical governance approaches

Exercises: Technical governance approaches

Resources: AI and the years ahead

Exercises: AI and the years ahead

Resources: Contributing to AI safety

Exercises: Contributing to AI safety

Resources: What is AI alignment?

Exercises: What is AI alignment?

Resources: Further developing your project

Exercises: Further developing your project

Resources: Scalable oversight

Exercises: Scalable oversight

Resources: Robustness unlearning and control

Exercises: Robustness unlearning and control

Resources: Reinforcement learning from human (or AI) feedback

Exercises: Reinforcement learning from human (or AI) feedback

**This course has been replaced by our new [Technical AI Safety course](https://bluedot.org/courses/technical-ai-safety).**

How and why might we build future AI systems?

We will first describe AI as a collection of many approaches. We will then examine crucial new additions to these techniques: neural networks, gradient descent and transformers. We'll look at how these techniques are used to train large language models (LLMs), the technology behind most state of the art foundation models like ChatGPT.

We’ll then explore what exactly AI companies are trying to build, and why companies seem to be racing to develop and release these services.

By the end of the unit, you should be able to:
\- Describe a framing for thinking about AI and its impacts
\- Explain the impact of advances in hardware technology
\- Define key ML terms including weights, biases, parameters, neurons and activations
\- Explain at a high level how neural networks are trained with gradient descent
\- State the structure of a trained large language model (LLM), and explain the base and fine-tuning LLM training stages
\- Understand economic and non-economic incentives behind developing transformative AI systems
\- Describe key advances in AI capabilities over the past few years


AI systems today are remarkably good at many tasks, though there are still many domains in which they cannot yet outperform humans.

In this exercise, you'll consider how much of your day-to-day work could be replaced or augmented by an AI system today, and how that might be different in 10 years time. You'll use these insights in your cohort's discussion to explore how AI could impact all of society.

(If you're not currently employed, either consider your most recent job, or the next job you think you'll have)

1. What type of tasks comprise most of your working hours? Take note of the different categories. How well do AI systems currently perform at those tasks?
2. Now imagine it's 2034. Based on the rate of improvement demonstrated in the resources, what might AI systems look like in 10 more years? How much better are our AI systems that can input and output [images](https://www.midjourney.com/showcase), [videos](https://www.youtube.com/watch?v=NXpdyAWLDas), [audio](https://www.udio.com/) and [text](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/#context-window) data (including [code](https://www.cognition.ai/blog/introducing-devin))? What long [sequences of actions](https://www.multion.ai/api) can they take via the internet - and maybe even in the [real world](https://www.youtube.com/watch?v=-e1_QhJ1EhQ)?
3. How much of your day-to-day work do you expect would now be possible by AI systems? How much of your friends and family's work will be affected?
4. How do you think this will affect your life, in terms of the work you're doing? If you'd find a different job, what jobs do you think might exist? If not, what would you do instead?


How will AI affect your job in 10 years time?

> Software is eating the world

Incredible advances have taken place throughout the economy due to the development of sophisticated software systems. However, these systems are often too complex for any single human to understand, and faults that develop into significant problems are often missed. This can happen even when everyone is acting with the best of intentions, or can be exacerbated when some actors are behaving maliciously or deceptively.

In this exercise, you'll research a specific case study where a fault in a complex software system led to significant harm. In your cohort's session, you'll use these findings to explore implications and learnings for AI development.

Instructions
- Choose a case study from the list below, or find your own case study  
- Tell your cohort (in Slack) what case study you've selected - ideally so you can cover a range of case studies
- Research the case study and answer the prompts below. We recommend spending 15-30 minutes on this.

Case studies
- [British Post Office scandal](https://www.wikiwand.com/en/British_Post_Office_scandal)
- [Uber self-driving car crash](https://www.wikiwand.com/en/Death_of_Elaine_Herzberg)
- [Boeing 737 Max](https://www.justice.gov/opa/pr/boeing-charged-737-max-fraud-conspiracy-and-agrees-pay-over-25-billion)
- [Volkswagen emissions scandal](https://www.wikiwand.com/en/Volkswagen_emissions_scandal)
- [Therac-25 Radiation Therapy Machine](https://www.wikiwand.com/en/Therac-25)
- [London Ambulance Service software failure](https://web.archive.org/web/20231020230153/https://bedford-computing.co.uk/learning/wp-content/uploads/2016/09/Report_London_Ambulance_Service-Jan2015.pdf)
- [Disinformation on Facebook and the Rohingya genocide](https://www.youtube.com/watch?v=K8B0bWO9u3M)
- [Toyota 'Unintended Acceleration’](https://abcnews.go.com/Blotter/toyota-pay-12b-hiding-deadly-unintended-acceleration/story?id=22972214)

(feel free to use other resources to find out more about your case study!)

Prompts
1. What was the purpose and use-case of the software system?
2. What were the faults, and how severe were the consequences?
3. What other factors contributed to the incident?
4. How did the incentives of the people involved affect the emergence of the fault?


Why complex systems often fail

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below. 

1. In a neural network, a \_\_\_\_\_ is a number that defines how much one neuron should influence another.
2. In a neural network, a \_\_\_\_\_ is a number that adds a constant amount to a neuron's activation.
3. What's the value of A, B and C in [this neural network](https://docs.google.com/drawings/d/1raj4TGZG_F1mPOUKawHKiS-DaGULNFodjK-bTzxf5IE/edit) that uses the [sigmoid function](https://www.wolframalpha.com/input?i=sigmoid%283.14%29)?
4. What does the cost function represent when training a neural network?
5. Why does minimising the cost function improve the neural network's performance?
6. How does gradient descent work - i.e. minimise the cost function? (at a high level)
7. Does gradient descent guarantee finding the best model? Why or why not?
8. Explain how this except from our website might be turned into data to train an LLM: “BlueDot Impact was founded in August 2022 in Cambridge, UK.”
9. What are the two main stages Karpathy splits up training an LLM into?
10. What three factors have contributed the most to AI improvements over the last few years?
11. Compared to Chess and Go, AI systems improving at games like Dota 2, Minecraft and Diplomacy show greater ability to work in what kinds of environments?
12. Why might people be incentivised to deploy AI systems that pursue goals and make plans?
13. As well as foundation models, narrow models such as DeepMind’s protein folding model called \_\_\_\_\_\_ can significantly advance life sciences.
14. What type of systems have OpenAI and Anthropic stated they intend to build?

You can check your answers against [the answer key](https://docs.google.com/document/d/1Z3D3tfVvoMXynRh4PbZ6hSL5vm0OAg6vCtOL9N96l6E/edit).


(Optional) Comprehension questions

Get a feel for how current AI models work! In this exercise you'll try out a large language model (LLM), and optionally a text to image model.

You can write down your notes and answers to the exercises in the box below.

**Part 1: Large language model**

LLMs take text as input, and output more text. They're commonly used for tasks like chatbots. To complete this exercise, you'll need access to one. We recommend [Bing Chat](https://www.bing.com/chat) because it's free and doesn't require signup, but you can use any chat model you'd like such as [OpenAI's ChatGPT](https://chat.openai.com/), [Anthropic's Claude](https://claude.ai/chats), [Google Deepmind's Gemini](https://gemini.google.com/app), [Inflection's Pi](https://pi.ai/talk), or [Meta's Llama 2](https://replicate.com/blog/run-llama-locally).

Ask your chatbot for a recipe for your favourite food, but delivered as a rhyming poem.

1．What did it do well? What did it seem to struggle with?

Now ask it how gradient descent works, summarised in a few sentences. Ask it follow up questions if you want more information. (Tip: You can use AI models while on the course to help understand resources - but be aware that they sometimes make mistakes!)

2．Does this align with your understanding from the resources?

Now, ask your chatbot what '13943642 × 1343' is. Compare this result with a calculator.

3．Did it answer correctly? Why might it be difficult for an LLM without tools to answer these kinds of questions?

**Part 2: Text to image model**

Text to image models take text as input, and output an image. To complete this exercise, you'll need access to one. We recommend [Bing Image Creator](https://www.bing.com/images/create) because it's free, but you can use any image generation model you'd like such as [OpenAI's Dalle-2](https://labs.openai.com/), [Midjourney](https://docs.midjourney.com/docs/quick-start) or [Stable Diffusion](https://clipdrop.co/stable-diffusion).

Ask your model for a picture of your favourite animal, playing your favourite sport on the moon.

4．What did it do well? What did it seem to struggle with?

You can compare your results against [our sample results](https://docs.google.com/document/d/11v2UxgUNOanub4DWiSIoOb5A_892zxX_K-HmDPuMXW8/edit#heading=h.qrtutjqgmx4v).


(Optional) Use some AI models

These questions build upon what you've learnt from the resources, but will also require you to search for new information e.g. on Google. As well as reinforcing the concepts you've learnt, they allow you to practice finding answers to new questions - important in a field like AI where things are advancing rapidly!

1. As of April 2025, what is NVIDIA’s best chip available for sale to train AI systems?
2. How much does an NVIDIA B200 chip cost, roughly?
3. How many B200 chips have X AI said they plan to purchase?
4. Who created the Falcon LLM?
5. Which country is this organisation based in?
6. What sizes (in terms of parameter counts) does the Falcon model come in?
7. Why might someone choose a version with more parameters?
8. Why might someone choose a version with fewer parameters?
9. What dataset was the model trained on?
10. Uncompressed, how big is this dataset? (e.g. in GB or TB)

You can check your answers against [the answer key](https://docs.google.com/document/d/1DnLR9OQU46Dhjj0zu3ImRKfEU8pUUDdfaDBJEnBP13s/edit).


(Optional) Do your own research

These exercises allow you to practice the maths behind reinforcement learning, gradient descent and backpropagation.
- [Exercises](https://docs.google.com/document/d/1frrCVxYNzHrhD54bwzuZkAQH9aauV6sQGG3hLJ2kZaM/edit)
- [Answer key](https://docs.google.com/document/d/1SR2ydpOIDGcRKGIUqFxLDqbXYecXnXrvOqzAIyuHsUU/edit#heading=h.313oaiblxob8)


(Optional) Bonus maths exercises

Pick a [UN sustainable development goal](https://sdgs.un.org/goals) (or get a [random one](https://wheelofnames.com/6uw-tpk)). A relevant 5 minute read: [The world is awful. The world is much better. The world can be much better.](https://ourworldindata.org/much-better-awful-can-be-better)

1. How has the world been performing on this goal so far?
2. How could transformative AI directly and indirectly support achieving this goal?
3. What factors, behaviours or developments would increase the likelihood of achieving this goal using AI?

Pick an existing and anticipated AI risk (maybe from [this list](https://aisafetyfundamentals.com/blog/ai-risks/)).

4．For the existing risk, find and summarise a case study not mentioned in the readings.

5．For the anticipated risk, find and summarise some related research or related case study not mentioned in the readings.

You can compare your answers against [our ](https://docs.google.com/document/d/1_JrtLuQY7l2NhbR2sT_Ktf_cUH6OWotda29sDaVBMps/edit)[example](https://docs.google.com/document/d/1OZAGhxb0QYjPKrNqIvquL_qlcGJ1nL_DP4WGEJL6DGA/edit)[ answers](https://docs.google.com/document/d/1_JrtLuQY7l2NhbR2sT_Ktf_cUH6OWotda29sDaVBMps/edit).


(Optional) Further investigating benefits and risks

AI Alignment

How and why might we build future AI systems?

AI and the years ahead

What do we need to do to ensure AI systems do what we want, and why is this difficult?