Prioritise writing and sharing

How to write well

How to share your work

Get it working

Figure out what needs to be true

Deciding what to test first

Using your results

Evaluation criteria

Submit your project

Certificate

You can make AI systems safer

Getting started

Project ideas

Scope it down

Get funding for your project

Here’s how you can start to make (perhaps your first!) AI safety research or engineering contribution in 30 hours. You don’t need a PhD or deep ML expertise, just familiarity with coding and a desire to make the future with AI go well.

Previous participants have completed projects like:
- Implementing a deception eval using UK AISI’s Inspect framework. ([see here](https://bluedot.org/projects/implementing-a-deception-eval-with-the-uk-aisis-inspect-framework))
- Reproducing METR’s findings on reward hacking in their research engineering benchmark. ([see here](https://fuzzyhead.substack.com/p/reproducing-metrs-re-bench-reward))
- Fixing open issues in TransformerLens. ([see here](https://medium.com/@anthonyduong1/my-ai-alignment-project-fixing-open-source-issues-25e59d32a16a))
- You can see other projects [here](https://bluedot.org/projects).

Many of our past graduates have found their co-founders, collaborators, roles and funding opportunities from their projects.

Our past graduates have also found these projects useful for:
- Improving their portfolio to help them gain AI safety opportunities 
- Testing their fit for particular types of AI safety work
- Upskilling in using a particular framework or library
- Understanding existing challenges in AI safety research

## **Who is this project sprint for?**
[You should apply](https://web.miniextensions.com/2lQmY04m6DdfeGbIYgsG) for the facilitated version if:
- You are comfortable with coding. (e.g. you have a few code projects or code professionally.)
- You have taken our [Technical AI Safety](https://bluedot.org/courses/technical-ai-safety) course (or equivalent)
- You want to test your fit for AI safety research / engineering OR you want to start building your portfolio.
    - If you’ve already been doing AI safety research / engineering, you might find this too beginner for you. 
- You are committed to spending at least 30 hours on this project.
- Not sure if you're a good fit? Apply anyway!
    - Note: Coding and research aren't the only ways to contribute. Check out our other courses, like the[ AGI Strategy](https://bluedot.org/courses/agi-strategy) and[ Biosecurity](https://bluedot.org/courses/biosecurity) course, for other ways to contribute.

## What to expect
- **Mentorship**: Weekly group check-ins and follow-ups with an AI safety expert who'll guide you on what's worth working on, what tools will help, what goals to set, and who to talk to.
- **Accountability**: [Setting goals is hard](https://xkcd.com/874/), sticking to them is even harder. Many past participants find collectively working towards a similar goal incredibly motivating!
- **Rapid** **feedback**: Having a point person to ask for feedback means faster learning and more iteration.
- **Project funding.** Once accepted onto the project sprint, you can [apply for a small grant](https://blog.bluedot.org/p/rapid-small-grants-for-the-bluedot) for compute or other project resources.

**You can also start right now.** You don't need to wait to be accepted onto the project sprint. Use this as a guide, and reach out to your BlueDot facilitator, the BlueDot community, or someone you trust for feedback. The structured course makes it easier — but the work is yours to begin whenever you're ready!


Part of what it takes to contribute to technical AI safety is developing good judgment (or “taste”) for what to work on. This includes what to work on, how to test your assumptions quickly, which results matter, and how to communicate your findings.

You build taste through exercising and improving your judgment. This sprint is your first guided rep of this practice, which you’ll complete with guidance from more experienced people in the field. 

The goal isn’t to produce something groundbreaking – it’s to start developing that practice.

### **How the sprint works**
The process follows a rhythm of narrowing and widening. 

In Unit 1, you'll narrow from the broad AI safety landscape to a specific paper, codebase, or project idea, and choose a direction to pursue. You're not expected to come up with something novel from scratch — you're building on other people's work.

In Units 2–3, you widen again: generate lots of ideas for what to test or build, then narrow by identifying which matter most and running cheap tests on them. Each time you test, reflect, and update your direction, that's one rep. 

You're encouraged to do as many reps as you can — everyone will move at a different pace, and that's expected!

In Unit 4, you write it up. In Unit 5, you share it.

Your mentor is there to help you make better decisions: whether your direction is tractable, whether you're testing the right things, and when to go deeper versus change course. **We recommend scheduling a 30 min call with your mentor 1-2 times a week.** 

### **Keeping a project log**
We strongly recommend keeping a log throughout the sprint. Use it to note your reasoning, what evidence changed your thinking, and how your predictions compared to actual results. 

Many AI safety researchers point to it as one of the most effective ways they developed their judgment early in their careers.

By the end of the sprint, you will have produced:
- **A project log** that you keep updated throughout the sprint.
- **A public write-up** that says: here's why I chose to work on this, here's what I tried and what I found, here's what I’d recommend pursuing next and why
- **A GitHub repository** with code that others can run, replicate or extend.


Set the permissions to “Anyone with the link can comment”, so your mentors can leave feedback.

You can use [this template](https://docs.google.com/document/d/1EG3l0-M9nJ1Mu7jLaCemnU1PL1rCmo8F89gvX-xSf1s/edit?tab=t.0).


Link to your research log

For example: “I am going to work on my project every day for 3 hours at 5pm while sitting at my living room table.”

Block this time in your calendar. Many projects fail because they weren’t prioritised. Let’s see this through to the end!


Write yourself a promise statement 

Before the first group discussion, choose a paper, codebase, or project idea to build on. Come prepared to explain why it matters for AI safety and share initial ideas for how you'd extend it. Your group and mentor will help you figure out which directions seem tractable and impactful.

Two types of projects you can pick:
- **Research:** replicate a finding and add a simple extension
- **Engineering:** improve or write code that makes it easier for others to do AI safety work (e.g. building tooling, resolving open issues)

Note: Though most people tend to gravitate towards research, don’t discount the usefulness of engineering. Well-written code can massively uplift researcher productivity and scale research. It also doesn’t require ML upskilling.

Here are some heuristics when coming up with project ideas:
- **Follow your curiosity! What piqued your interest in the Technical AI Safety course?**
- **Simple is fine.** They are easier to extend and explain.
- **Someone’s done it before?** Good! Follow their lead. Then, extend their work.
- **Picking a paper?** Make sure code and data are available. (You can do hard mode next time!)
- **Picking a repo?** Choose one with open issues you can start immediately.

Below are concrete project ideas to get you started. Some have been proposed directly by other organisations. You can also find open research questions and agendas in the optional resources section.

---
### **Contribute to Inspect Evals**
[Inspect Evals](https://github.com/UKGovernmentBEIS/inspect_evals) is the world's largest repository of open-source benchmarks. It is a platform for publishing and maintaining AI evaluations to serve the AI safety & security ecosystem. 

The platform includes: 
- **90+ open-source evaluations**, with an API that allows them to be seamlessly run via the CLI or python
- **Standards and guidelines** which promote development and implementation of best practices
- **Tools and supporting infrastructure** which streamline quality assurance and ongoing maintenance

Inspect Evals is used by:
- **Governments** during pre-deployment testing, which directly informs policymakers.
- **AI safety researchers.** E.g. UK AISI uses it to develop agent scaffolds which are used in their safety testing.
- **Orgs monitoring the trajectory of AI capabilities** (e.g , [Epoch AI dashboard](https://epoch.ai/benchmarks) and the [HAL Leaderboard](https://hal.cs.princeton.edu/))

You can contribute to their [open issues](https://github.com/UKGovernmentBEIS/inspect_evals/issues) following their [contributor guide](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/CONTRIBUTING.md), such as:
- **Implementing benchmarks:**
    - Taking benchmark code from a research paper and implementing it using Inspect to make it easier to re-run
    - Porting benchmarks from other frameworks (like lm-eval-harness) to Inspect
    - Adding new model provider support to existing evals
- **Building tools:**
    - Improving or extending [eval-specific scanners](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/AGENTS.md#check-agent-trajectories-inspect-scout) to improve error detection 
    - Tools that estimate the cost of running an eval with different models
    - Validation tools that check if scanners are catching what they should catch
- **Improve documentation and onboarding:**
    - Writing tutorials for specific Inspect features or patterns
    - Creating example implementations that demonstrate best practices
    - Improving setup/configuration documentation

_This project was suggested by the [AI Safety Engineering Task Force](https://www.linkedin.com/company/ai-safety-engineering-taskforce/)._

### **Grok’s MechaHitler incident: emergent misalignment in the wild?**
On 21 June 2025, [Elon Musk tweeted](https://x.com/elonmusk/status/1936493967320953090?lang=en) out a request for politically incorrect but true facts to train Grok. 3 weeks later, [Grok was calling itself MechaHitler](https://archive.ph/YktYp). What happened?

Could training the model on the politically incorrect data have caused it to learn seemingly unrelated misaligned behaviours? If so, this could be an example of [emergent misalignment](https://arxiv.org/abs/2502.17424) in the wild, where training on one type of problematic content (like insecure code) leads to unrelated misaligned behaviors (like suggesting violence).

In this project, you’ll scrape tweets to construct the dataset of politically incorrect facts, finetune a model on the tweets, then explore ways in which the resulting model is more misaligned. 

You may also want to explore:
- _[Subliminal learning](https://arxiv.org/pdf/2507.14805)_ — the finding that models can inherit hidden traits from training data even when that data seems unrelated to the trait. 
- Could other personas can be induced from finetuning on tweet-length beliefs?
- Investigate why the model generalized to calling itself Mecha-Hitler, rather than something else. Do persona-trained LLMs regularly maintain a self-conception as an AI?

_This project was suggested by FAR AI. Check out their open roles [here](https://www.far.ai/careers)._

### **Reinforcement learning with verifiable rewards (RLVR) side-effects**
RLVR trains models using rewards that can be automatically checked (e.g. math, coding) rather than human preferences. The concern is that optimising for "get the right answer" might cause models to develop unintended behaviours (e.g. deception or goal persistence) that help them succeed but weren't explicitly trained. 

There’s some evidence this may be happening. A [2025 Feb paper](https://arxiv.org/abs/2502.12206v1) claims that reasoning models exhibit greater [instrumental convergence](https://en.wikipedia.org/wiki/Instrumental_convergence) tendencies than non-reasoning models. 

AllenAI has since [released](https://huggingface.co/allenai/Olmo-3-32B-Think) a fully open reasoning model including checkpoints during RLVR. This offers an opportunity to observe interesting trends that might help us forecast potential risks from increased RL.

Some research questions you might want to pursue:
- Do the RLVR checkpoints show the instrumental convergence trends the paper predicts when evaluated on the [InstrumentalEval](https://github.com/yf-he/InstrumentalEval) benchmark?
- How do the learning dynamics for this misalignment compare to capabilities on their [MATH benchmark](https://huggingface.co/datasets/allenai/RLVR-MATH)? E.g., does instrumental convergence only occur once the capabilities benchmarks are saturated?
- How does the model change pre and post RLVR outside the training domain, like personality or more general reasoning ability? E.g. when you chat with it pre and post RLVR.

_This project was suggested by FAR AI. Check out their open roles [here](https://www.far.ai/careers)._

### **Beneficial capability evals**
Most evals on AI model cards measure dangerous capabilities. These help establish how worried we should be. But we're missing benchmarks for things we _want_ models to be doing. 

**Beneficial capability evals give AI companies something to aim for.**

If we're headed towards a capability explosion, we need to ensure beneficial capabilities will beat dangerous ones by a wide margin. 

Building beneficial capability evals also helps train safer models. An eval asks “can the model do X?” and produces a score. An RL environment asks the same question, but feeds that score back as a training signal. This means an eval is most of the way to an RL environment. And AI companies pay for RL environments.

Read more details on how you can get started [here](https://blog.bluedot.org/p/give-ai-companies-something-to-aim?_gl=1*1oulvwq*_gcl_aw*R0NMLjE3NjA3MzAwMjIuQ2p3S0NBancwc2ZIQmhCNkVpd0FRdHY1cWNlYmdqNkd5OG1OaEZDNTBoeTVRTHhLUVQzX3lWMVFfYU9pOWxKZHV5Q1I2UW9QYkJvQnd4b0NtWHNRQXZEX0J3RQ..*_gcl_ag*Mi4xLmswQUFBQUFfa1huWU4weU5SdlVOd1duN0Z5SGhaUER5YkFlJGkxNzYxNDIwMDc2JGJzTFpaQ09pZjBPNGFFUFBRb19ncCRiQUstekNKdVcyTWNhRUlPNzNzZ18kYm5rSE5DT3FmcnNvYUVQUFFvX2dwJGJXTTlVQ0xPYzBjd2FFUFBRb19ncCRiWGM2SUNPdWYwTzRhRVBQUW9fZ3AkYldNUjVDTUw3emNjYUVJTzczc2dfJGJMYjBQQ05LWnJzb2FFUFBRb19ncCRiSnB4dkNQMlZ6c2NhRUlPNzNzZ18kYm1pcS1DSWo4cmNvYUVQUFFvX2dwJGJHQnEwQ0tyQjZjTURFS3VONXBZQiRiSkJvOUNLM0I2Y01ERUt1TjVwWUI.*_gcl_au*NzY4MDc2MTE3LjE3NjU3NDEyNzEuMTA3MzIxMTc4OC4xNzY4NDUzMjMwLjE3Njg0NTM0Mzg.).

_This project was proposed in collaboration with [Adam Jones](https://adamjones.me/) from Anthropic._

### **Replicate and extend Anthropic’s alignment faking demo**
> We recently released a [paper](https://www.anthropic.com/research/alignment-faking) presenting a demonstration of alignment faking where we do not give the model any explicit goal, and instead study whether alignment faking can emerge due to the model's prior harmlessness training.
> 
> [This post](https://alignment.anthropic.com/2024/how-to-alignment-faking/) describes how to get started with experimenting with this demonstration yourself, and presents some ideas for future research.

They extended this work in [this blog post](https://www.lesswrong.com/posts/Fr4QsQT52RFKHvCAH/alignment-faking-revisited-improved-classifiers-and-open) and [paper](https://www.lesswrong.com/posts/ghESoA8mo3fv9Yx3E/why-do-some-language-models-fake-alignment-while-others-don), which also studies and open-source fine-tuned models.


For the paper, codebase or project idea that you’d be excited to build, explain the following in your own words: 
- What problem is this work trying to solve, and why does it matter for AI safety?
- What approach did the authors take, and why did they make those choices over alternatives?
- What are the key assumptions this work rests on? What would change if any of them were wrong?

Aim for <200 words. If you can't explain it simply, you probably don't understand it well enough yet.	


Understanding your base

Pick your top 3 most interesting directions for extending this work. For each, write one sentence on why pursuing it could be useful.

You'll practice explaining your reasoning to peers who don't know your project, and they'll test whether your explanation holds up and whether your directions follow from the base work.


Generate directions

Many technical projects require access to compute (e.g. renting GPUs) or expensive API credits to use frontier models. We want your project to be excellent, and don’t want money to be a barrier for you building your portfolio. 

**If your project will benefit from funding, [apply to our rapid small grant](https://airtable.com/appMVNtdBtvtJvu5E/pag9G3oF4DYAyassX/form).**

## **How it works**
1. **Submit a proposal [here](https://airtable.com/appMVNtdBtvtJvu5E/pag9G3oF4DYAyassX/form).**
    1. We typically respond within 5 working days with one of three outcomes:
        1. _Accepted_: We’ll confirm exactly what spending we’ve approved.
        2. _Clarification needed_: We’ll ask follow-up questions to evaluate your request.
        3. _Not approved_: We don’t think this meets our criteria, or we’re unable to approve for another reason.
2. **Do your project**, confident you can spend on what we’ve approved.
3. **Claim reimbursement.** We process most claims within 5 working days. If you didn’t end up spending the money, no problem - just don’t submit a claim.

Assume we’ll cover costs we’ve confirmed in writing for your specific project. If you’re uncertain, contact us before spending money you expect us to reimburse.

## **What we fund**
The following examples are illustrative:
- **Compute** for a technical AI safety project (API costs, cloud GPU, training runs).
- **Access to paywalled resources** like articles, research papers, datasets, or textbooks.
- **Hosting costs** for your application or tool.

We expect most grants to fall between **$50-100** for initial experiments to develop your proof of concept, then up to **$500** once you have evidence of initial traction (a strong proof of concept or promising preliminary results). We encourage you to submit a separate application for each of these stages.

## **What we don’t fund**
- **Compensation for your time** on the project.
- **Equipment** you’d reasonably already have (laptops, phones, external drives, etc.)
- **General productivity subscriptions** (ChatGPT Plus, Claude Pro, Cursor, Grammarly, etc. unless this is highly leveraged).
- **Personal expenses.**

Other funders may cover some of these. See [this resources page for AI safety funding opportunities](https://www.aisafety.com/funding).

## **When should you apply?**
Working through the exercises in this unit and engaging in your first group discussion should give you a clear enough sense of your project idea to start applying.

It doesn’t need to be a slam dunk, but you should:
- Be able to briefly articulate how it relates to AI safety.
- Have ideas for initial experiments you want to run.
- Have a rough estimate for how much those experiments will cost (I recommend using an LLM to help you calculate this).

If you’re unsure whether your application is strong enough, submit an application anyway! 

While you wait for a grant decision, you should continue to develop your project idea, do a deeper literature review, or start to run smaller experiments that fit on the free [Google Colab](https://colab.research.google.com/) GPU.

## **Eligibility**
- You must be a current or past participant of BlueDot Impact’s Technical AI Safety Project Sprint.
- We reimburse via bank transfer (Wise) or PayPal (UK), so we cannot send payments to sanctioned countries.
