Resources: Foundations of AI Governance

Exercises: Foundations of AI Governance

Resources: The AI Policy Toolkit

Exercises: The AI Policy Toolkit

Resources: How AI Systems Work

Exercises: How AI Systems Work

Resources: Proposals to Govern AI

Exercises: Proposals to Govern AI

Resources: The Promise and Perils of AI

Exercises: The Promise and Perils of AI

Resources: Contributing to AI Governance

Exercises: Contributing to AI Governance

Next steps: Programs

Resources: How AI is Governed Today

Exercises: How AI is Governed Today

Resources: Governance at Frontier AI Companies

Exercises: Governance at Frontier AI Companies

This unit will focus on the potential positive and negative effects of AI in the present and future. First, we’ll explore the concept of transformative AI and discuss the potential AI might have to transform our society. We’ll examine the promise of AI systems to boost productivity, automate scientific progress, and make systems more efficient. We’ll also take a look at what the companies building AI are saying about the potential for their systems to change the world.

Then, we’ll turn to AI’s potential perils. Many leading AI scientists and entrepreneurs have called for mitigating “the risk of extinction from AI” to be “a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Whether or not AI’s harms escalate to extinction, AI could contribute to global-scale misuse, conflict, and accidents.

The field is far from having consensus on the nature, scale, or likelihood of different anticipated benefits and risks. Still, responsible policymaking should be informed by the reasons why many experts are hopeful, and why many are concerned. The nature of today’s capabilities – and how they might lead to tomorrow’s promises and perils – is the focus of this unit's resources.

Note that due to this unit’s focus, many of the resources come from individuals and groups who are directly involved in the development, deployment, and implementation of AI systems, or those who are incentivizing such work. We think it is important that policy professionals understand what these actors are saying about their work and what incentives drive them to pursue it. You should keep this in mind when evaluating the merits of their claims.

It’s extremely important to finish all of the resources in this unit. It’s hard to know what goal your policymaking should aim for without understanding what today’s technology might be able to do in the future.

By the end of the unit, you should be able to:
\- Define and explain transformative AI and alignment
\- Evaluate the potential benefits AI systems might offer 
\- Identify the incentives, motivations, and goals that drive actors to build and deploy AI systems
\- Describe how today’s AI risks might scale in the future as AI systems improve
\- Explain why some experts believe advanced AI might constitute an existential risk to humanity


_Spend 15-30 minutes writing your response._
In this exercise, you'll imagine how AI might change a specific sector.
First, pick a sector from the list below.
- Healthcare
- Finance
- Transportation and logistics
- Manufacturing
- Retail
- Technology
- Entertainment
- Media and journalism
- Customer service
- Education
- Other (feel free to choose a sector not on this list)
For your chosen sector, create an evidence-based prediction of how this sector might be positively impacted by AI progress by 2034, ten years from now. 
As you prepare to write your prediction, consider researching from sources like:
- Forecasts about general AI capabilities and capabilities
- Expert opinions
- The existing state of AI in your chosen sector
- Whether your sector is likely to adopt new technology quickly or slowly
Once you’ve completed your research, spend 10-15 minutes writing your prediction in the box below. Be sure to cite your sources and explain your reasoning. 


Imagining sector-specific AI futures

_Spend 30 minutes writing your response_
For this exercise, read one of the existing AI risk case studies below. Tell your cohort (in Slack) what case study you've selected - ideally so you can cover a range of case studies.
1. [Malfunctions](https://web.archive.org/web/20240416093654/https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html)
2. [Discrimination](https://www.wired.com/story/wrongful-arrests-ai-derailed-3-mens-lives/)
3. [Reducing social connection](https://web.archive.org/web/20240814072317/https://www.businessinsider.com/when-your-ai-says-she-loves-you-2023-10)
4. [Surveillance](https://web.archive.org/web/20240419051341/https://www.nytimes.com/2019/04/14/technology/china-surveillance-artificial-intelligence-racial-profiling.html)
5. [Warfare](https://www.972mag.com/lavender-ai-israeli-army-gaza/)
6. [Disinformation](https://www.wired.com/story/slovakias-election-deepfakes-show-ai-is-a-danger-to-democracy/)
7. [Automation](https://web.archive.org/web/20240411011132/https://www.nytimes.com/2024/04/10/business/investment-banking-jobs-artificial-intelligence.html)
After you read, conduct some research outside your case study to get a sense of how pervasive this risk is right now and how this risk might get worse in the future (i.e. in the next decade). We recommend finding 2-3 additional sources. Consider using a chatbot with internet access to help you.
Now, spend 15-30 minutes responding to the prompts below. Write your responses in the text box provided.
1. Describe the AI system that enabled the risk in your case study
2. How widespread does this risk seem to be? Is this an isolated incident, or is this risk spreading?
3. As AI develops over the next decade, how could this risk get worse?



AI risks in the world today

_Spend 10-20 minutes on this exercise_
In this exercise, you will review what a major company pursuing advanced AI systems says about building generally intelligent AI. First, pick one of the companies below and click the link to read or watch something they’ve produced about what advanced AI might look like:
- [OpenAI](https://openai.com/blog/planning-for-agi-and-beyond)
- [Anthropic](https://www.anthropic.com/news/core-views-on-ai-safety) (read the introduction and “Our Views on Rapid AI Progress”)
- [Google DeepMind](https://deepmind.google/discover/blog/real-world-challenges-for-agi/)
- [Meta](https://www.threads.net/@zuck/post/C2QBoRaRmR1)
Then, evaluate your company’s claims. Write down:
- The key claims about AI's potential benefits
- The evidence or reasoning (or lack thereof) provided for these claims
- Any potential counterarguments or uncertainties
Finally, answer the following prompt:
Based on what you’ve just read, is the company you chose pursuing a clear vision of a positive AI future? Can you concretely explain the positive benefits this company hopes to achieve with advanced AI – or is it too vague?


Evaluating what AI companies say they are trying to achieve

_Spend 15 minutes writing your response_
This exercise aims to help you consolidate your understanding of how AI might exacerbate long-running catastrophic risks, and the novel threats introduced by this session's resources.
**Do you think anticipated risks arising from misuse, accidents, or rogue, agentic AI systems, are more likely to cause harm?**
**Does this answer change when limiting your time horizon to 5 years, 15 years, and 30 years?**
You may want to consider:
- Whether you feel these are **useful categories** of risk.
- The **scale of the harm caused** by each scenario.
- How you think each risk might **concretely manifest as an actual scenario**.
If you are not compelled by any of the anticipated risks outlined in this week’s resources, or from your own experience, feel free to structure your essay around reasons why you believe they won't materialize.


Understanding anticipated AI risks

AI Governance (2024-2025)

The Promise and Perils of AI

How AI Systems Work

Foundations of AI Governance

The AI Policy Toolkit

How AI is Governed Today

Governance at Frontier AI Companies

Proposals to Govern AI

Contributing to AI Governance