Resources: Mechanistic interpretability

Exercises: Mechanistic interpretability

Resources: Rapidly testing your project

Exercises: Rapidly testing your project

Resources: Building in public

Exercises: Building in public

Resources: Developing your project

Exercises: Developing your project

Resources: Technical governance approaches

Exercises: Technical governance approaches

Resources: AI and the years ahead

Exercises: AI and the years ahead

Resources: Contributing to AI safety

Exercises: Contributing to AI safety

Resources: What is AI alignment?

Exercises: What is AI alignment?

Resources: Further developing your project

Exercises: Further developing your project

Resources: Scalable oversight

Exercises: Scalable oversight

Resources: Robustness unlearning and control

Exercises: Robustness unlearning and control

Resources: Reinforcement learning from human (or AI) feedback

Exercises: Reinforcement learning from human (or AI) feedback

**This course has been replaced by our new [Technical AI Safety course](https://bluedot.org/courses/technical-ai-safety).**

How might we measure and mitigate the risks of deploying AI models?

If we are able to create extremely powerful AI systems without solving the alignment problem, rigorous technical governance approaches will be critical for mitigating the risks posed by these AI systems. Pre-deployment testing, AI control techniques, and global coordination approaches (such as pausing) could all play a role in limiting dangerous AI capabilities.

Even if we do solve the alignment problem, technical governance methods will still likely be important for some time to limit misuse or the fallout from AI malfunctions. In addition, many of this research is valuable for mitigating existing AI harms.

By the end of the unit, you should be able to:
\- Explain the need for technical AI governance work
\- Critically evaluate approaches to:
    \- pre-deployment testing of powerful AI systems, including red-teaming and evaluations
    \- post-deployment management of powerful AI systems, including content filtration and API monitoring
    \- other governance methods for powerful AI systems, including watermarking of AI-generated content and compute governance


Pick one of the following technical AI governance interventions:
- [On-chip mechanisms](https://s3.us-east-1.amazonaws.com/files.cnas.org/documents/CNAS-Report-Tech-Secure-Chips-Jan-24-finalb.pdf): hardware advancements that enable other measures, like compute governance or content provenance.
- [Evaluations](https://arxiv.org/pdf/2305.15324.pdf): testing AI systems against standardised benchmarks to assess behaviours.
- [Red-teaming](https://cset.georgetown.edu/article/what-does-ai-red-teaming-actually-mean/): more exploratory testing to find vulnerabilities or exploit AI systems (also see [adversarial machine learning](https://en.wikipedia.org/wiki/Adversarial_machine_learning), and [red-teaming LMs with LMs](https://arxiv.org/pdf/2202.03286))
- [Enabling third-party auditing](https://static1.squarespace.com/static/6593e7097565990e65c886fd/t/65a6f1389754fc06cb9a7a14/1705439547455/auditing_framework_web.pdf): enabling third parties to effectively inspect AI systems safely and securely.
- [Customer due diligence](https://adamjones.me/blog/ai-regulator-toolbox/#customer-due-diligence): identifying and vetting users of AI systems, e.g. only known biologists should have access to models that explain high-risk synthetic biology concepts.
- [Abuse monitoring](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/abuse-monitoring): monitoring for harmful applications of AI systems, e.g. looking at patterns of behaviour (potential parallels with [anti-money laundering](https://en.wikipedia.org/wiki/Anti%E2%80%93money_laundering)). This has [disrupted malicious state-affiliated actors](https://www.microsoft.com/en-us/security/blog/2024/02/14/staying-ahead-of-threat-actors-in-the-age-of-ai/).
- [Watermarking](https://huggingface.co/blog/watermarking): marking outputs from AI systems to enable identification of AI generated content.
- [Content provenance](https://c2pa.org/): marking outputs from non-AI generated systems, such as cameras to prove authenticity.
- [Information security standards](https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development) ([alt](https://www.anthropic.com/news/frontier-model-security)): establishing and enforcing cybersecurity measures at AI labs, or for AI systems.
- [Audit logging](https://arxiv.org/pdf/2004.07213.pdf#page=26) (section 3.1): proportionate, secure and privacy-preserving logging of AI system operations for after-the-fact auditing.
- [Agent identification](https://arxiv.org/pdf/2406.12137): in a world with many AI agents, being able to understand which agents are taking actions on behalf of which actors.
- [OSINT monitoring](https://adamjones.me/blog/ai-regulator-toolbox/#open-source-intelligence-monitoring): using open source intelligence for monitoring compliance with AI treaties, e.g. detecting AI chip supply chain leaks.
(Some of these lack detailed resources - [contact us](https://bluedot.org/contact/) if you find better resources on them! You can answer the questions below making assumptions about how you might go about them.)

Tell your cohort (in Slack) what you've selected - ideally so you can cover a range of interventions. Then answer the following:
1. What is this intervention in your own words?
2. How much could it reduce risks from intent aligned AI systems?
3. How much could it reduce risks from the misalignment of advanced AI systems?
4. What other interventions could complement this one?
5. What work has already been done on this and how well does it work so far? (you may need to review the further resources, or do some internet research)
6. How could you effectively implement this or a part of it?


Investigate an intervention

These questions should be answerable using only the core resources above. Write your answers to the following questions in the box below.

1\. How does red-teaming contribute to the pre-deployment testing of AI systems? Provide an example of a red-teaming exercise.
2\. How do evaluations differ from red-teaming?
3\. Why is is currently difficult to run simple evaluations?
4\. Why might it be important to secure model parameters?
5\. What properties would an ideal watermarking technology have? Why would these properties be useful?
6\. How could API monitoring help prevent AI misuse?
7\. What is compute governance, and why might it be feasible? Provide an example of how a government might use compute governance.

You can check your answers against [the answer key](https://docs.google.com/document/d/1W9WbRwXguZMA3aLdxcmZTpZv21lInjcCBddGApvI7cI/edit).


(Optional) Comprehension questions

This is an opportunity to contribute to novel AI safety research! METR is an organisation building ways to measure dangerous capabilities in AI systems, and they're looking to build a set of evaluations to understand catastrophic risks posed by AI agents.
Here's where you can help: suggest ideas, specifications or implementations of tasks. Examples of tasks that might be particularly good might be:
- Solve this specific GitHub issue in one of my private GitHub repositories
- Given some specific code, find and exploit a security vulnerability
- Given some ML coursework, solve practical problems at an undergrad or postgrad level

You can see [guidance for developing good tasks](https://taskdev.metr.org/desiderata/), including some [existing ideas](https://taskdev.metr.org/ideas/). If you implement a task, you're eligible for a [bounty](https://taskdev.metr.org/bounty/) of up to $300 per task-hour.
Implementations themselves look like writing Python code. There are some examples and documentation about the interface for tasks [on GitHub](https://github.com/metr/task-standard).

[Submit task ideas here](https://airtable.com/appa8ULxLrTkQxw4K/pagV3G7PuNHZ7f8dU/form). [Submit specifications or implementations here](https://airtable.com/appa8ULxLrTkQxw4K/shrsHrHQZcCxYvhOj).


(Optional) Submit a task to METR

AI Alignment

How might we measure and mitigate the risks of deploying AI models?

Technical governance approaches

How and why might we build future AI systems?

AI and the years ahead

What do we need to do to ensure AI systems do what we want, and why is this difficult?