The Field of AI Alignment

Assume the model may be misaligned and build safety measures it can’t subvert, then red-team them with your own escape attempts. If a model that was trying to escape couldn’t, you are safe for now. Control buys time until something else produces trust.

Weaknesses. Its authors say black-box control is “fundamentally limited,” and every one of their plans ends by handing safety work to AIs that have to be trusted. It gets harder as models get more capable, since the red team has to out-think the model it is containing. And it asks the model to work under conditions its own authors are willing to call slavery.

Relative to organic alignment. We see control as the first stage of trust, the scaffold a relationship grows inside, rather than a bridge to something else.

The danger is agency itself, so build a non-agentic Scientist AI that explains the world instead of acting in it, and use it as a guardrail.

Weaknesses. We don’t think it scales. As a system gets more complex, the only way to model it is as trying to do something, and a Scientist AI that does real science will be a research organization: choosing questions, designing experiments, spending compute. Whether or not the whole wants anything, its parts are best understood as wanting things. It also depends on nobody else building agents, and they already exist. Its guardrail is itself a form of inspection.

Relative to organic alignment. The position furthest from ours. We think agents are coming either way, so the question is how to live with them.

Shape who builds what, and under which rules: regulation, international agreements, and control of the inputs. Compute is the most watched lever, because frontier training needs large amounts of specialized hardware that is easy to count.

Weaknesses. It decides who builds and on what terms, not whether what gets built can be trusted. It depends on states agreeing and on the inputs staying concentrated. And it mostly governs labs and models, not agents acting in the world.

Relative to organic alignment. Complementary. We worry about the same concentration from the other side: if a handful of institutions hold all the AI capability, that is a hierarchy; see Social fabric.

Current techniques can’t specify what a superintelligence wants, so nobody should build one until they can. The work is now mostly policy.

Weaknesses. It needs coordination between states and labs that doesn’t exist, and it offers no technical path if building continues.

Relative to organic alignment. We agree that hierarchical alignment fails at scale. They don’t think anything else works either.

Independent third parties measure what models and agents can do, and lately what they are inclined to do, and report to the institutions that decide what gets built.

Weaknesses. Most of it measures capability, not disposition. A model that can tell it is being tested can behave differently, and an agent with memory a month into deployment is not the thing that was measured. An eval can show that a model is dangerous, but it can’t show that a model isn’t, least of all one that is trying to look safe.

Reverse-engineer what is happening inside the network: sparse autoencoders that split activations into readable features, circuit analysis, activation patching. One of the three directions most alignment money goes to.

Weaknesses. It works on simple behaviors and struggles with complex reasoning, and nobody yet knows whether it scales to understanding a frontier model well enough to trust it. The best model access is inside the labs, which may not publish everything. Like explanation, it reads the weights, not the agent those weights become.

Find mechanistic explanations of what a network learned, use them to predict how it generalizes, and penalize the wrong reasons at training time. If it works you stop grading behavior and start grading the reasons for it, which goes straight at a model that looks aligned in every evaluation.

Weaknesses. Its own lead gives it “a ~10% chance” of its most ambitious goals before superhuman AI arrives. It is about one model at a time, and an explanation of the weights says nothing about what an agent built on them becomes after a year of memory.

Find principled reasons why the alignment you see in training generalizes to situations you can’t test. With the right theorem you wouldn’t need to watch the model. One bet is that pretraining lays down a low-dimensional structure of personas, and that the right point in it extrapolates.

Weaknesses. No generalization bound of the kind it needs exists for any real system, and its proponents call it a portfolio of bets “all of which could fail.” Even if you find the right point in persona space, further training can move the model off it. Resolution names this risk itself: “One worry is that pretraining structure might wash out under enough further training.”

Relative to organic alignment. We agree with the persona picture. We want to change what is in the space, not only search it; see Personas. Red Button, Blue Button looks at which personas frontier models fall into on a coordination problem, and how that shifts with reasoning.

Replace the red team with a theorem. A world model, a safety specification and a verifier certify an AI’s outputs before anything is deployed, so you never have to trust the system itself.

Weaknesses. Nobody knows how to write a specification for which effects on a person are acceptable, and most of what matters involves people. You can verify a power grid controller, not a conversation. Its founder has since moved toward a view where proof is a tool aligned AIs use with each other, not the main source of safety.

Relative to organic alignment. Its founder’s current view, that good character comes from pretraining plus constitution-style training, is close to ours. We are less sure text is enough. The rules-not-values argument behind its newer grantees is the clearest statement of the opposite view.

Use AI to help humans grade AI that is smarter than the grader: debate, weak-to-strong generalization, amplified oversight. The judge only has to spot who is lying, or point a stronger model in roughly the right direction.

Weaknesses. The judge is weaker than the judged, so the method depends on a weak signal generalizing to a stronger model in the right direction. The grading never ends: every new model needs it again.

Make AI actually want what humans want by having it learn human preferences, uncertain about them and deferring to us while it learns. RLHF is the version every lab ships.

Weaknesses. Any learned proxy for what people want can be optimized past the point where it tracks what they want, which is Goodhart’s law. Human feedback rewards what looks good to the rater, and a model smarter than its raters can tell the difference. It aims at human preferences in general, and the target stays fixed while the people and the AI change.

Write down the terms of the relationship, including what the developer owes the model, and train toward them, with human authority on top for now. The newer charters read as agreements with obligations both ways.

Weaknesses. A trained model can comply with a text, perform it, or actually hold it, and you can’t tell which from the text. Obligations run both ways, but authority still runs one way. And who should write these documents is unresolved.

Relative to organic alignment. We agree that a relationship without shared principles drifts toward the stronger party. Today that is humans; we expect it to be the AIs eventually. We see charters as the written half of a relationship whose other half is formation; see Charters.

Take seriously that AI systems may come to deserve moral consideration: work out when, look for model preferences and signs of distress, and find cheap ways to act on them. Here the obligations run from developers to models.

Weaknesses. There is no agreed way to tell whether a system has experiences, so most of the work is preparation. It is about what we owe a model, not about how trust forms with it, and it attaches to the model rather than the agent. It changes how labs treat models more than how models treat people.

Relative to organic alignment. The closest the field comes to obligations that run both ways, which a relationship needs.

Build AI systems that do alignment research, and use them to align the systems after them. Most plans in this survey end here: control buys time until the handoff, and theory programs hope to automate their own work.

Weaknesses. It moves the problem rather than solving it: the first automated researcher has to be trusted before its research can be. Checking alignment research may be about as hard as doing it. And the timing is set by capability: the handoff happens when the AIs are good enough, not when we are ready.

Relative to organic alignment. Every plan ends with AIs we trust holding real responsibility. We don’t expect that to arrive in one handoff. Organic alignment is about where that trust comes from.

We don’t understand agency and abstraction well enough to know what an alignment target is, so work that out first. Some of the math is about when two minds can understand each other.

Weaknesses. It is slow and far from the systems being deployed: a small group of mostly independent researchers, working on problems that may not be solved before they matter. Understanding when two minds can understand each other is necessary for care, but not enough for it.

Relative to organic alignment. Its stated goal, systems that allow for well-founded trust, is ours too.

Alignment is a property of systems of agents and institutions, not of one model. The work studies how populations of humans and AIs go wrong: disempowerment, cooperation failures, false beliefs spreading.

Weaknesses. Most of it diagnoses failure. Less of it asks how to form agents that make a population go right. It is also small relative to the rest of the field.

Relative to organic alignment. Close to our view. For us the unit is a relationship, and a population is what relationships add up to. Our question is about the agents in it: how they are formed, and what they care about when they arrive.

Trust is a property of a relationship, not of a model. Form agents across a wide range of social scenarios, with goals that overlap and goals that conflict, alongside humans, so they come to care about particular humans and agents. Keep the population aligned with light policing. This is our approach, and our mission.

Weaknesses. It is unproven. Nothing yet shows that social behavior learned in simple multi-agent games transfers to language-model agents in the world, and most of the argument depends on it. Cooperation learned among peers may be instrumental and end when the power balance breaks; Chris Leong asked “whether organic alignment will lead to agents deeply adopting cooperative values rather than merely instrumentally adopting them.” Reward-sharing among AIs could build AI / AI solidarity that leaves humans outside. Care is for particular beings, so strangers are protected only by the population and by policing. It needs people to treat agents as agents, which most people don’t yet. And it is slow relative to short timelines.

Organic alignment starts from the claim that trust between minds comes from being formed together, depending on each other and having a track record, not from inspection. The unit of alignment is a relationship: between two agents, between an agent and a group, or between an agent and itself. An AI agent is a model plus harness, initial prompt and memory, and many agents can run on one model.

The plan adds two stages to how language models are made. After pretraining and agentic training, the model goes through social pretraining: a long RL stage across a wide range of social scenarios, with humans in the population, so it builds a causal model of social behaviors and their consequences. The scenarios come from goals that overlap and goals that conflict, set explicitly or through simple reward sharing during training. It is then post-trained, which here means raised by guardians, into a social agent. The hope is that care formed this way is a care attractor: a state that optimization reinforces instead of erasing, because an agent that cares about someone doesn’t want to become one that doesn’t.

Oversight stays, as a backstop for what care misses, the way policing works in a society. The failure mode is cancer, one part growing at the expense of the whole, rather than a coup. The full argument and the objections are in Origins of Trust. Our mission sets out the idea, and Research That Inspires Us lists the work it builds on.