The Science of Alignment

Lead: Zach Furman

Alignment in practice consists of making interventions to the training setup (pretraining data, RL environments, etc), but this only affects the behavior of AI models through the training process. Studying how, when, and why capabilities emerge during the training process would better allow us to predict and control the behavior of AI systems.

Lead: Stephan Wäldchen

Alignment in practice has so far relied on RLHF to reward good behaviour. But how can a weaker human judge reliably evaluate a superintelligence? Prover-verifier games capture this for formal domains like mathematical proofs, and we aim to extend them to include statistical reasoning about the natural world. One promising approach is Debate, where two AI debaters recursively narrow their disagreement until it is simple enough for a human judge. We study when such protocols make honesty the unique equilibrium.

Lead: George Robinson

Our team works on heuristic arguments and mechanistic explanation. We look for examples in mathematics where phenomena can be explained by structures, especially structures that arise through training. It’s a broad-ranging agenda centred on finding structure in algorithms, which lets researchers bring expertise from their own fields to AI safety.

Lead: Louis Jaburi

Models generalize during training. To understand and steer that process we would like to trace certain model behavior back to the training data. Which seemingly innocuous data points are actually malign? We develop methods that can answer this question in a reliable and tractable way. This requires understanding the training process and leveraging this knowledge to understand at what point capabilities are acquired and how we can influence that process.

Leads: Daniel Chiang, Dalcy Ku, Satya Benson

We study how agents form concepts about the world, how they should pursue goals with limited computation, and how goal-directed behavior arises in physical systems. We assume only what should hold for any capable agent, including one that has redesigned itself, so our results don't depend on today's architectures. We want to understand what it would take for an AI system's goals to refer to what humans care about, and how we could check that a candidate system's goals would stay that way as it grows more capable. Current work includes mathematical theories of when different minds can share concepts, of which goals and beliefs survive self-modification, and of what makes a physical system an agent at all.

Lead: Arturs Berzins

The Structure of Neural Networks group aims to advance learning theory and bridge it to interpretability. We want to deeply understand the space and organization of neural network solutions, how it is shaped by the data and the architecture, and what a learning algorithm selects. Our goal is to connect this geometric structure to safety-relevant observables and establish principled ways to control the selection of solutions.