Power, Windows, and Dependencies

Exercise: Map it out

Contested Questions in AI Governance

How should we handle open weights as models get more capable?

When, if ever, should frontier development be slowed - and by whom?

[wip] What even are the prospects for international AI governance?

[wip] How much direct control should different governments have over AGI?

[wip] How should we handle the geopolitical implications of catastrophic dual-use capabilities? 

Tracking Hub

Finding Your Leverage Point

More Career Resources

Next Steps: Fellowships

Certificate

State of Play at the Frontier

Evaluating Claims

The Evals Theory of Change

METR

Epoch

The System Card

Beyond Corporate Promises

Exercise: Briefings

Grappling with the Intelligence Explosion 

Reframing the debate

Exercise: Governance Under Acceleration

Visions of Victory

Face off

Argue it out (with yourself)

Optional but highly recommended

Unit 5 will force you to go deep on one of AI Governance's key contested questions. 
Groups are able to vote on which question they are most interested in and focus on that question in their session - see the other chunk titles for things we may cover. 


The relevant question used to be "should we be open?" - kind of like asking whether we should connect our very best AI models to the internet. We all know how that went.

But that question is no longer live. Open models are now genuinely good, and there are a lot of them. Beyond the models themselves, we're seeing open training code, data, RL environments, and checkpoints. Training 70B+ models on decentralized consumer hardware is feasible. With a single H100, you can run plenty of models downloaded from HuggingFace - where you'll also find lovely abliterated and unsafe variants. That said, open models do provide real economic benefits, enable privacy-conscious adoption, and draw power away from the frontier triplets (OpenAI, Anthropic, Google DeepMind).


1. State your position. Choose an actual position on the spectrum and be specific about where you sit and why!
2. State your level of uncertainty. What are you least confident about? What would change your mind? What's the strongest argument against your position?
3. Check coherence with your strategic vision. Maybe go back to the strategic visions table. Does your position on this question match what your chosen vision implies?



Write ~1 page on where you stand!

The relevant question used to be "should we pause AI?" 

That question generated a famous open letter, thousands of signatures, and not much change. Even the people who signed it couldn't agree on what they meant - true of many community letters today as well. Even in a full-on debate week (back when your course lead was taking v1 of this course) participants who all believed AI might destroy the world couldn't agree on whether a pause would help reduce the risk of that. 

Almost three years later, the debate has hardened into a loop: the pause advocate proposes a bilateral agreement; the opponent yells about unilateral surrender to China; the advocate explains, again, that it's bilateral; the opponent yells louder about defection costs. Meanwhile, the only lab that had a somewhat binding pause commitment - Anthropic - dropped it in February 2026, concluding that unilateral restraint in a competitive market is strategically irrational... And the same week, the Pentagon threatened to invoke wartime production law to force unrestricted military access to Claude.

So this naive question is well dead. But the harder questions underneath it are more alive than ever. What would a pacing mechanism actually look like - could if-then commitments at capability triggers do the job? Do we need an international treaty with compute monitoring (backed by what technology)? A moratorium enforced by airstrike? Nukes? Who would operate it? Could you verify compliance? And is it even worth trying - given that delay has an opportunity cost measured in cured diseases, scientific breakthroughs, and ultimately lives? 


1. State your position
    1. Not just "slow down" or "don't slow down." Pick a mechanism - or argue against all of them. The MIRI treaty? If-then commitments? Something else? Or write an update on whether Belrose is right and restraint backfires, and the safety-optimal strategy is continued iterative development? How optimistic should we be about alignment? Or on if Bostrom is right and only minimal slowdowns of a few years are worth the cost? 
    2. Be specific about what you would actually build, what kind of world it leads to, who would operate it, and what you'd do when it breaks.
2. State your level of uncertainty. 
    1. Carlsmith says restraint is logically necessary if you take the risk seriously. Bostrom says the math often doesn't support long slowdowns. Belrose says restraint degrades the alignment research it's supposed to buy time for. The MIRI treaty depends on compute monitoring that Carlsmith himself notes can't track algorithmic progress... Anthropic couldn't sustain even a voluntary conditional mechanism. 
    2. Which challenge most undermines your position? What would change your mind?
3. Check coherence with your strategic vision. 
    1. Your Unit 3 position implies a view here (maybe you've even updated on it since). If you favour strong governance, do you trust the same institutions that dropped the RSP under competitive pressure - or that invoked wartime law for unrestricted AI access - to operate a pacing regime? If you believe alignment is hard and timelines are short, Carlsmith's logic says you should want restraint - but Belrose says restraint might make alignment harder. 
    2. Can you hold both? Does your Unit 4 reading on race dynamics and power concentration make any restraint mechanism look like a tool for concentration rather than safety?


Frontier AI Governance

The Proposal Landscape

Race Conditions, Power Concentration and Governance in the Limit