Resources: AI standards and regulations

Exercises: AI standards and regulations

Resources: Introduction to Machine Learning

Exercises: Introduction to Machine Learning

Resources: Career Advice (AIG)

Exercises: Career Advice (AIG)

Resources: Closing regulatory gaps through international agreements

Exercises: Closing regulatory gaps through international agreements

Resources: Projects  (AIG)

Exercises: Projects  (AIG)

Resources: Challenges in achieving AI safety

Exercises: Challenges in achieving AI safety

Resources: Additional proposals

Exercises: Additional proposals

Resources: Introduction to potential catastrophic risks from AI

Exercises: Introduction to potential catastrophic risks from AI

Resources: Closing regulatory gaps through non-proliferation

Exercises: Closing regulatory gaps through non-proliferation

Last week’s readings introduced many AI risks. This week, we will dive into some nuances of AI safety (especially AI alignment), which is often especially poorly understood. In particular, this week’s readings focus on why AI safety may be challenging to achieve—even if most AI developers act responsibly. 

By the end of the unit, you should be able to:
\- Explain why AI systems might behave differently in testing than in deployment. (That would be a challenge for achieving safety through testing.)
\- Describe a few mechanisms by which future, widely capable AI agents could hypothetically take power from humans. (If that escalated sufficiently, it could be irreversible.)
    \- (Why this matters: the prospect of irreversible damage from AI misalignment or misuse raises the stakes, and it suggests it may be unwise to just plan on learning from failures.)
\- Describe a few challenges faced by research agendas in AI safety.
\- Identify several social/political dynamics that may make it harder to achieve AI safety.


Read[ our guidance](https://docs.google.com/document/d/e/2PACX-1vTPYrsCZV95S03L5-cO-p10cHz1TBMxSr_onylcSdO7KHNVmBSeOGVTq1LDlO49tXHoAJ_oZh6D4pt1/pub) on how to approach writing to learn exercises.
If you work in AI governance, it’s likely you’ll be bridging the gap between technical and social challenges in AI development and public policy. Therefore it’s important to understand the underlying risks.
In this exercise, you will practise making a case for addressing an AI risk to an audience (in this case elected representatives) who have multiple competing priorities. 

**For this exercise, write an _introduction_ to a policy recommendation **(note you won’t be making an actual policy recommendation, you’re just writing a background / motivation section to an imaginary policy recommendation)**. Pick _one_ of the following risks or challenges that your recommendation is addressing:**

**_Technical problems:_**
- Power-seeking behaviour
- Deceptive behaviour post-deployment
- Difficulty in correctly specifying outcomes humans value in a loss function (e.g. reward misspecification and [Goodhart’s law](https://en.wikipedia.org/wiki/Goodhart%27s_law))
- Goal misgeneralisation [from optional readings]
**_Incentive Structures:_**
- A ‘Race to the Bottom’ on AI safety
- Competitive pressure to delegate power to AI systems
- Acceleration of AI research by AI systems
**_Or choose your own_**

**Your response should:**
- Explain the technical source of the risk such that a non-technical reader feels they understand.
- Motivate the need for the policy, by explaining what outcomes the risk might entail.
To reiterate, you don’t need to come up with an actual policy recommendation for this exercise.
When choosing a topic, you may also want to consider:
- Which key points and risks from these readings would you prioritise, and why?
- What other priorities might your audience have (e.g. economic incentives, balancing risks and opportunity of AI, etc). How can you address their concerns, or reassure them about their priorities?
- Feel free to further specify the purpose and audience of the policy document, if it helps you to narrow down the scope.


Challenges in achieving AI Safety

We include these prompts to help give you ideas about questions for further investigation. Use them as you wish.
**Prompts:**
- What do the actions of non-human agents (e.g. companies and governments) imply for how likely we are to be able to address ‘alignment’ and ‘power seeking’ in (hypothetical) AI agents? In what ways do these types of agents differ from AI agents?
- What methods are employed by other governance structures you’re aware of, to scrutinise and immobilise deceptive agents? Are any relevant to deceptive AI agents, and what methods can we employ against AI deception that don’t work in other fields?
- Are there fields with similar challenges as those posed in "AI Safety Seems Hard to Measure"? What mechanisms have they used to circumvent those challenges?
- Are there any aspects of AI safety that you believe could be more measurable than others, and if so, why?
- In your opinion, how justified is the concern raised in "Nobody’s on the ball on AGI alignment" about the size of the AI safety field and the challenges it faces? How would you propose we overcome these challenges?
- If you were to advise policy makers on AI safety, what key points from these readings would you emphasize and why?
- How do these readings influence your perspective on the role of competition in AI development? Could competition potentially speed up progress towards safe AI, or is it more likely to exacerbate the risks?


AI Governance (2023)

Exercises: Challenges in achieving AI safety

Exercises