Detection overview

Detection deep-dive

Option 1: Country-specific biosurveillance

Option 2: Metagenomic sequencing technology roadmap

Option 3: Rapid individual testing

Intro - Delay patient zero

Delay overview

Delay deep-dive

Option 1: Deter - Reducing intent

Option 2: Deny - Regulating discovery

Option 3: Deny - Implementing AI model-level safeguards

Option 4: Deny - Safeguarding research infrastructure

Intro

The history of pandemics and bioweapons

The future of pandemics and bioweapons?

Towards a pandemic-proof world

[Optional] Biology 101 for biosecurity

[Optional] Biosecurity overviews

Rapid MCM overview

Rapid MCM development deep-dive

Option 1: Faster R&D

Option 2: Clinical trials innovation

Option 3: Better financing and procurement

Option 4: Faster manufacturing and delivery

Your next steps

Choose your focus

Biosecurity project ideas

Create your 1-pager

Next steps: Technical fellowships

Next steps: Policy fellowships

Next steps: Career resources

Next steps: Programs

NPI overview

NPI deep-dive

Option 1: Pandemic-proof PPE

Option 2: Biohardening indoor spaces

Option 3: Social interventions

**The best way to limit damage from a pandemic is to prevent it from happening in the first place.**

In this unit you'll explore interventions to prevent engineered pandemics, either through accidental release from a lab or a deliberate biological attack. We choose this focus because there is evidence that biorisk posed by advances in AI and synthetic biology have the potential to be substantial, are changing rapidly and there are tractable interventions to address them.

We don't explore the prevention of naturally-occurring pandemics (e.g. disease spillover from animals to humans) in this section. If you want to learn more about this, [this paper](https://www.nature.com/articles/d41586-022-01312-y) is a good place to start. The material in later units on detection and response apply to all pandemic threats, regardless of origin.

We'll start this unit by exploring what AI models can and cannot currently do, how we try to measure this and how rapidly these capabilities are improving.


Work in biosecurity is intended to prevent harm from pandemics. Because we know that any one defense layer may fail against a determined adversary, we rely on stacking many independent layers. 

In medicine, we call this approach the [Swiss cheese model](https://upload.wikimedia.org/wikipedia/commons/b/bf/Covid-19-Cheese-Model-animation-02-short.gif) - a threat that passes through a hole in our defenses should, in expectation, be stopped by a subsequent defense layer.

**Biosecurity as a whole can be conceptualized as a series of defensive layers, namely**
1. Delay (or entirely prevent) the creation of engineered pandemics
2. Detect pandemics early to buy time for ramping up defense
3. Withstand the initial shock of a pandemic
4. Combat the pandemic and ultimately eradicate it

**The Delay layer, which is the focus of this session, can itself be broken down into sub-layers**
1. Deter adversaries from pursuing the development of biological weapons 
2. Deny adversaries access to the knowledge and physical infrastructure needed to develop and produce a pandemic
3. Intercept adversaries in the process of developing or producing a pandemic



Pick 1 of the following 4 interventions to dive deeper into. We've given you some resources on the following pages as a starting point.

Aim to answer the following general questions and have a look at the deep dive pages for more detailed, intervention-specific questions:
- What are the intended goals of this intervention?
- What is the current state of this intervention, and to what extent is it achieving its goals?
- What are the most critical holes in this intervention?
- What are they key challenges to implementing a more robust version of this intervention, and how could you overcome them?

You will bring the results of your research to share in your group discussion so make sure to take notes and share great explainers or visualizations you have found.

**Option 1: Deter - Reducing intent**
Disincentivizing the development of biological weapons.

Topics you will research:
- The Biological Weapons Convention (BWC)
- Genetic engineering attribution 
- Open Source Intelligence (OSINT)

**Option 2: Deny - Regulating discovery**
Some knowledge becomes dangerous the moment it is created, so society needs to decide what is researched and who should be allowed to access the findings.

Topics you will research:
- The de-facto ban on mirror life research
- 'Dual-Use Research of Concern' (DURC), 'Gain-of-function' (GoF) research and 'Pathogens with Enhanced Pandemic Potential' (PEPP)
- Virus hunting programs


**Option 3: Deny - Implementing AI model-level safeguards**
General Purpose AI (GPAI) beats experts on knowledge tests and state-of-the art Biological AI Models (BAIM) make biology increasingly programmable. Preventing bad actors from using them to do harm while preserving their usefulness for beneficial research requires good safeguards.

Topics you will research:
- Pre-training filtering
- Fine-tuning + weight locking
- Output filtering

**Option 4: Deny - Safeguarding research infrastructure**
A genome sequence is only a threat if someone can turn it into a functional virus. We can prevent this by blocking key chokepoints.

Topics you will research:
- Nucleic acid synthesis
- Know your customer (KYC) regulation
- Cloud labs and Contract Research Organizations (CROs)





Delay patient zero: Deep-dives

**Disincentivizing the development of biological weapons.**

Interventions falling under deterrence span:
- resolving the international conflicts that motivate state biological weapons programs, 
- strengthening the Biological Weapons Convention (which currently has no verification mechanism and fewer than 5 permanent staff)
- building attribution capabilities, i.e., the forensics that allow identifying the source of (engineered) outbreaks credibly
In a time of renewed geopolitical competition deterrence is challenging but has high upside potential.

**Topics on which you could focus while researching your deep dive**
- The Biological Weapons Convention (BWC)
- Genetic engineering attribution 
- Open Source Intelligence (OSINT)

**Guiding questions**
_Note: The questions are meant to show relevant search direction, you don’t have to answer any or all of them_
- Why does the BWC not have a verification mechanism?
- How exactly would attributing a virus to a country or lab work? What did historic attempts at this look like?
- How do OSINT groups build credibility?



**Some knowledge becomes dangerous the moment it is created.**

Scientific innovation always comes with dual-use concerns:
- The [Haber-Bosch](https://en.wikipedia.org/wiki/Haber_process) process enabled massive agricultural yield gains but also made the production of military explosives possible at large-scale 
- [Organophosphates](https://en.wikipedia.org/wiki/Organophosphate) are highly effective agents for pest control, but are also the [most potent chemical weapon class](https://en.wikipedia.org/wiki/Nerve_agent) known to this day
- Industrial-scale fermentation is a boon for food production but also allowed the Soviet bioweapons program to rapidly produce [large amounts](https://pmc.ncbi.nlm.nih.gov/articles/PMC10134958/) of anthrax 

Regulating innovation restricts open science and academic freedom by design. The case for banning mirror life research is unusually clear, but the discussion around other research areas, e.g., gain-of-function research is ongoing. 

Broadly, interventions aimed at denying adversaries the knowledge to do harm fall into three categories:
1. Regulating knowledge generation (i.e., what gets researched)
2. Regulating knowledge documentation (i.e., what gets published)
3. Regulating knowledge access (i.e., who can use the knowledge)

**Topics on which you could focus while researching your deep dive**
- De-facto ban on mirror life research
- 'Dual-Use Research of Concern' (DURC),  'Gain-of-function' (GoF) research and 'Pathogens with Enhanced Pandemic Potential' (PEPP)
- Virus hunting programs

**Guiding questions**
_Note: The questions are meant to show relevant search direction, you don’t have to answer any or all of them_
- What was the case for pursuing mirror life research? Are there groups still researching mirror life? 
- What is the process that individual scientists use to determine the risk-benefit ratio of their research?
- What were the benefits of virus hunting programs? Why were they shut down?



**General Purpose AI beats experts on knowledge tests and state-of-the art Biological AI Models make biology increasingly programmable.**

General Purpose AI (like LLMs) provides new ways to aggregate and explain dangerous information and to draw non-obvious conclusions. Every major AI developer now treats biological uplift as one of the risks they must actively mitigate. So far, LLMs do not provide meaningful uplift to [novices](https://activesite.substack.com/p/rct), but this could rapidly change as frontier AI companies are taking massive bets on integrating AI into the scientific discovery process.

The risk from GPAI is compounded by recent breakthroughs in Biological AI Models. BAIM make it possible to understand and manipulate biology with increasing sophistication, which increases the possible ceiling for biological harm.

The overlap between this deep dive and classic AI Safety work is considerable and many interventions are being carried over directly.

**Topics on which you could focus while researching your deep dive**
- Pre-training filtering
- Fine-tuning  + weight locking
- Output filtering
- API control
- Proof of personhood

**Guiding questions**
_Note: The questions are meant to show relevant search direction, you don’t have to answer any or all of them_
- Which interventions produce the most robust denial for threat actors while preserving utility for legitimate researchers?
- Why is training models to refuse harmful biological outputs hard?
- What are examples of failed approaches to safeguarding GPAI and BAIMs and why did they fail?


**A genome sequence is only a threat if someone can turn it into a functional virus, blocking key chokepoints prevents that.**

Nucleic acids (DNA and RNA) define the blueprint for humans and other animals, but also of viruses, bacteria and other pathogens. A bad actor could come up with a dangerous sequence on a computer, but it would need to be 'printed' (synthesised) to cause harm in the real world. 

Interventions designed to make it harder for threat actors to produce and weaponize pathogens include preventing access to 
- Pathogen sequences (e.g. DNA synthesis screening)
- Lab equipment (e.g. special fermentation gear)
- Contract research organization and cloud labs

**Topics on which you could focus while researching your deep dive**
- Nucleic acid synthesis
- Know your customer (KYC) regulation
- Cloud labs

**Guiding questions**
_Note: The questions are meant to show relevant search direction, you don’t have to answer any or all of them_
- What is the difference between gene synthesis, DNA synthesis and nucleic acid synthesis?
- What are common, legitimate use cases for nucleic acid synthesis?
- KYC protocols are common in the financial services sector - why are they not common in biotechnology?
- How dangerous are cloud labs?



Biosecurity

Option 3: Deny - Implementing AI model-level safeguards

Resources (50 mins)