Power, Windows, and Dependencies

Exercise: Map it out

Contested Questions in AI Governance

How should we handle open weights as models get more capable?

When, if ever, should frontier development be slowed - and by whom?

Finding Your Leverage Point

More Career Resources

Next Steps: Fellowships

Next steps: Programs

State of Play at the Frontier

Evaluating Claims

The Evals Theory of Change

METR

Epoch and Jagged Capabilities

The System Card

Independent Investigation: The Hugging Face Incident

Exercise: Briefings

Grappling with the Intelligence Explosion 

Reframing the debate

Exercise: Governance Under Acceleration

Visions of Victory

Face off: Plan S vs Plan A

Argue it out (with yourself)

Unit 5 asks you to take a defended position on one contested question in AI governance.

Your group chooses one of the two developed branches: **open weights** or **restraint of frontier development**. Complete the readings and approximately one-page position paper for your selected branch. The discussion then tests your position against concrete scenarios.


The relevant question used to be "should we be open?" - kind of like asking whether we should connect our very best AI models to the internet. We all know how that went.

But that question is no longer live. Open models are now genuinely good, and there are a lot of them. Beyond the models themselves, we're seeing open training code, data, RL environments, and checkpoints. Training 70B+ models on decentralized consumer hardware is feasible. With a single H100, you can run plenty of models downloaded from HuggingFace - where you'll also find lovely abliterated and unsafe variants. That said, open models do provide real economic benefits, enable privacy-conscious adoption, and draw power away from the frontier triplets (OpenAI, Anthropic, Google DeepMind).


1. State your position. Choose an actual position on the spectrum and be specific about where you sit and why!
2. State your level of uncertainty. What are you least confident about? What would change your mind? What's the strongest argument against your position?
3. Check coherence with your strategic vision. Maybe go back to the strategic visions table. Does your position on this question match what your chosen vision implies?



Write ~1 page on where you stand!

When should frontier development be slowed, what should be restricted, and who should decide?

Revisit your Unit 3 position after the acceleration analysis in Unit 4. Start with the short current-source readings below. Distinguish a request to build coordination tools, a conditional commitment, a reported operational restriction, and an enforceable policy. Identify what each source actually establishes and what remains uncertain.

Then compare the RSP and MIRI agreement with Carlsmith’s analysis. Specify the activities covered, the evidence that triggers action, the authority to decide, verification, exceptions, enforcement, and conditions for resumption. Ask what work and compute would move elsewhere under the restriction.

Test your preferred mechanism against Bostrom and Belrose: could it delay important benefits, obstruct safety work, create an overhang, or concentrate power? What useful work would the time bought make possible?


1. **State your position.** Choose a restraint mechanism, or defend continued iterative development. Specify the activity covered, the decision-maker, trigger evidence, verification, consequences of noncompliance, and conditions for resumption. Distinguish capacity you would build now from restrictions you would activate now.
2. **Test your mechanism.** Apply it to the developer-reported OpenAI case. What would it have restricted, what work could continue or move elsewhere, and what evidence would establish success? Name one claim the public account cannot establish. Explain what safety work or preparation the time bought enables, and confront the strongest Bostrom or Belrose objection.
3. **State your uncertainty and check coherence.** Identify the weakest dependency: verification, competition, institutional competence, concentrated power, forgone benefits, or safety progress. State what would change your recommendation. Can the institutions on your Unit 2 map operate it under Unit 4’s acceleration assumptions? Explain how this changes or supports your Unit 3 position. You may support neither Plan S nor Plan A.


Frontier AI Governance

The Proposal Landscape

Race Conditions, Power Concentration and Governance in the Limit

Frontier AI Governance

How should we handle open weights as models get more capable?

Resources (4 hrs 40 mins)

Exercises