[An 8-week part-time course](https://aisafetyfundamentals.com/alignment-course-details/) can't cover everything there is to know about [AI alignment](https://aisafetyfundamentals.com/blog/what-is-ai-alignment/), especially given it's a fast-moving field with many budding research agendas.

This resource gives a brief introduction into a few areas we didn't touch on, with pointers to resources if you want to explore them further.

## Developmental interpretability

We covered mechanistic interpretability (or "mech interp") [in session 5](https://course.aisafetyfundamentals.com/alignment?session=5). To recap, this is an approach that 'zooms in' to models to make sense of their learned representations and weights through methods like feature or circuit analysis.

Developmental interpretability (or "dev interp") instead studies how the structure of models changes as they are trained. Rather than features and circuits, here the object of study is phases and phase transitions during the training process. This takes inspiration from developmental biology, which increased our understanding of biology by studying the key steps of how living organisms develop.

[Read more](https://www.lesswrong.com/posts/TjaeCWvLZtEDAS5Ex/towards-developmental-interpretability).

## Agent foundations

Most of this course has explored fairly empirical research agendas that have been targeting the kinds of AI systems we have today: neural network architectures trained with gradient descent. However, it's unclear whether we'll continue to build future AI systems with similar technology. You'll also have likely realised that a lot of the fundamentals underpinning the field can be unclear: the field does not have an agreed definition of what alignment is, or the kinds of properties we might want from an aligned system.

Agent foundations research is primarily theoretical research that aims to give us the building blocks to solve key problems in the field, particularly around how powerful agentic AI systems might behave. It draws a lot from maths, philosophy, and theoretical computer science, and in some ways is the [basic science](https://en.wikipedia.org/wiki/Basic_research) of alignment.

[Read more](https://intelligence.org/files/TechnicalAgenda.pdf).

## Shard theory

Shard theory is a research approach that rejects the notion agents are following specified goals, and instead suggests that reinforcement learning agents are better understood as being driven by many 'shards'. Each shard then influences the model's decision process in certain contexts.

This takes inspiration from how humans appear to learn. Most people do not optimise some specific objective function all the time, but behave differently in different contexts: for example when their 'hunger' shard activates this might influence a decision to stop working and go for lunch, because this has worked well in the past (rather than because it optimises some complex objective function).

Viewed through this lens, shards appear to result in models choosing actions over others in different situations, which indicates the model's values. How shards develop therefore affects what values a model might also develop. Research in shard theory generally focuses on understanding the relationship between training processes and the resulting shards and learned values.

[Read more](https://www.lesswrong.com/posts/8ccTZ9ZxpJrvnxt4F/shard-theory-in-nine-theses-a-distillation-and-critical).

## Cooperative AI safety research

Most of the research we've looked at has focused on making individual AI systems safer. However, there are already many AI systems being created and used by different people. In future, these may be taking far more actions in the real world (see [Open Interpreter](https://twitter.com/OpenInterpreter/status/1770821439458840846) for an example).

We've previously seen instances of algorithmic systems interacting [in](https://www.technologyreview.com/2016/10/07/244656/algorithms-probably-caused-a-flash-crash-of-the-british-pound/) [undesirable](https://www.michaeleisen.org/blog/?p=358) [ways](https://en.wikipedia.org/wiki/2010_flash_crash), and [current AI systems seem to escalate conflicts when interacting with each other](https://arxiv.org/pdf/2401.03408.pdf). AI systems that are more capable and have control of more resources acting in undesirable ways might be a significant risk.

Cooperative AI safety research (also called multi-agent safety research) explores how we might avoid or mitigate the impacts of these negative interactions. It also explores how we might encourage agents to coordinate in ways that result in positive outcomes.

[Read more](https://arxiv.org/pdf/2012.08630.pdf).

## Model organisms of misalignment

We are already seeing [many AI risks](https://aisafetyfundamentals.com/blog/ai-risks/) starting to materialise. However, it's still unclear how anticipated risks will emerge, particularly risks stemming from deceptive misalignment. This is where models appear to be aligned (possibly during training, or when they're being carefully overseen by humans), but are actually misaligned and will pursue a different goal when deployed or not carefully supervised.

Better understanding how and why deceptive misalignment occurs would help inform how we might prepare for these risks. It could also help better-inform policy decisions about AI safety, as well as decisions about what AI safety research to prioritise.

Research into model organisms of misalignment seeks to achieve this by intentionally creating and then studying deceptively misaligned AI systems or subcomponents, in a controlled and safe way.

[Read more](https://www.alignmentforum.org/posts/ChDH335ckdvpxXaXX/model-organisms-of-misalignment-the-case-for-a-new-pillar-of-1).

## Technical moral philosophy

_This is a particularly vague area with limited research, and we're not certain whether this is the correct title or description for this research area._

We [briefly mentioned moral philosophy](https://aisafetyfundamentals.com/blog/what-is-ai-alignment/?_gl=1*1ctv0i3*_gcl_au*ODg4NzA2MDg0LjE3NDYwMjI0MDA.*_ga*NjYxMzA1MDQ1LjE3NDYwMjI0MDA.*_ga_LGQ75B1P00*MTc0NjExNDIwNS43LjEuMTc0NjExNjAzNi4wLjAuMA..#:~:text=Moral%20philosophy%3A,hard%20to%20reconcile.) earlier on the course, which we described as determining intentions that would be "good" for AI systems to have.

Technical moral philosophy research explores technical solutions or problems to identify good optimisation targets for advanced AI systems. Its more theoretical nature often means it overlaps with philosophy and agent foundations research.

Recent work includes [Anthropic's collective constitutional AI work](https://www.anthropic.com/news/collective-constitutional-ai-aligning-a-language-model-with-public-input), [OpenAI's democratic inputs to AI program](https://openai.com/blog/democratic-inputs-to-ai-grant-program-update), and [the Meaning Alignment Institute's work on moral graphs](https://meaningalignment.org/values-and-alignment-paper).

## Other training approaches

These approaches are fairly general approaches to training AI systems. However, they are potentially relevant to AI safety work.

[Imitation learning](https://ai.stanford.edu/blog/learning-to-imitate/) is an approach to training machine learning systems to copy humans. It is thought that this might result in better aligned AI systems because more capable systems might just copy humans better rather than get highly misaligned. It should be noted though that in some senses pre-training an LLM is imitation learning - to imitate what humans type on the internet - and this has not resulted in aligned AI systems.

[Inverse reinforcement learning](https://ai.stanford.edu/~ang/papers/icml00-irl.pdf) is an approach to machine learning that intends to infer an intended goal by observing human behaviour. This is similar to imitation learning but adds an extra step to infer the intended objective from the data (rather than matching the data being the objective itself).

Some safety research goes into seeing what can be done to make both of these approaches [more competitive](https://arxiv.org/pdf/2205.07886.pdf), [more accessible](https://arxiv.org/pdf/2211.11972.pdf) and more likely to result in aligned systems (for example, with [scalable oversight](https://aisafetyfundamentals.com/blog/scalable-oversight-intro/) techniques).

## Others?

There are likely other less well-known alignment research agendas that we've missed. You can find other new ideas discussed in the AI Safety Fundamentals Slack, or on online forums like the [AI Alignment Forum](https://www.alignmentforum.org/).


What we didn't cover in our June 2024 AI Alignment course

Adam Jones • August 13, 2024