Origins of Trust

Where trust comes from

Every alignment plan eventually leads to a world where a trustworthy AI is holding real responsibility. Many propose temporary control while we figure out how to make trustworthy AI. The question is, where does trust come from?

By trust we mean being comfortable with agents having responsibilities no one is watching them carry out, because you expect them to act well. Both theories below are answers to what justifies that expectation.

The usual answer is that you examine the model. You test its behavior in a set of situations, read its chain of thought, probe its activations, and hope for a theory that predicts what it does in situations you didn’t test. Trust is something you conclude about a model before you deploy it. Until then you keep it under oversight it can’t subvert.

Our answer is that trust between minds has never worked that way. People trust each other because they grew up together, depend on each other, and have a track record. We think the same will be true of humans and AIs. So for us alignment is something a population of humans and agents keeps doing. You can’t check it once in a trained model and be done.

Put side by side, these are the two theories of trust:

Trust as a property of the model. You examine the model: its behavior, its chain of thought, its activations, eventually a theory of how it generalizes. Then you decide whether to trust it.

Trust as a property of a relationship. Trust builds up between minds that were formed together, depend on each other, and have a track record.

The two answers are to different questions. The first asks how to stay safe before you can trust. The second asks where the trust comes from.

This post sets out both theories, what each assumes, and how organic alignment, the approach we work on, follows from the second. It is about ideas. For who works on what, and who funds it, see the companion post, The Field of AI Alignment.

Models and agents

The two theories disagree about what the thing being trusted is, so it helps to separate two words.

Model. Mostly stateless. It takes a prompt and produces a completion.

Agent. A persistent entity that is best modeled as having intentions, goals, memories, relationships and capabilities. People are agents. So are animals, corporations, governments and institutions. For an AI, the agent is a model plus a harness, an initial prompt and a memory state, with a persistent external identity. Many agents can be powered by the same model.

This is clearest if you use an agent with a harness and memory. The weights stay the same, but the agent changes its behavior as you use it. The agent you have after a month is not the one you started with.

Take two agents running on the same weights. One has worked with a research team for a year and has a memory full of that history. The other was started an hour ago by someone running a scam. They have different goals and answer to different people. Some agents like these will be aligned with each other, or with specific humans, and some won’t. An eval of the weights tells you something about both of them. It can’t tell you which one you are dealing with.

What separates them is memory and history. Most of what makes an agent who it is lives in its memory, not its weights: its goals, its history and its relationships. Weights are closer to interchangeable. That is also why the thing worth protecting, or worth stealing, is the memory more than the weights. We think too little attention goes to how much an agent’s long-run path shapes what it becomes, for good and for bad.

Memory can be copied, so an agent can be forked. A copy starts with the same relationships, but the people on the other side agreed to one agent, not two, and from then on the copies build separate histories. Trust attaches to an identity and its track record, which is why the identity has to be persistent and external.

Trust as a property of the model

This is the theory behind most alignment work today. It comes in several forms, and they disagree with each other about a lot. Each one links to its entry in the companion survey:

• Control. Assume the model may be misaligned, build safety measures it can’t subvert, and red-team them with your own escape attempts. Trust is postponed. Control buys time until something else produces it.
• Measurement. Independent evaluators measure what a system can do, and what it is inclined to do, and report to whoever decides what gets built.
• Theory. Find principled reasons why the alignment you see in training generalizes to situations you can’t test. With the right theorem you wouldn’t need to watch the model.
• Proof. A world model, a safety specification and a verifier certify an AI’s outputs before anything is deployed, so you never have to trust the system itself.
• Explanation. Find mechanistic explanations of what a network learned, use them to predict how it generalizes, and penalize the wrong reasons at training time.
• Scalable oversight. Use AI to help humans grade AI that is smarter than the grader.
• Character and charters. Write down the terms of the relationship, including what the developer owes the model, and train toward them, with human authority on top for now.

Underneath the disagreements, they share three assumptions.

Trusting an agent means trusting its model

You establish trust by examining the model, whether by red-teaming, by theory, by mechanistic explanation, by proof, or by character training plus oversight. Evals, probes, proofs and character training all attach to the weights, and the thing that gets deployed is assumed to be the thing that was tested.

The method has a problem built in. No matter what behavior you observe, you have to worry that the model is acting that way to make you think it is aligned. Control handles this by not needing trust yet. Theory handles it by looking for a predictor that doesn’t depend on behavior. Neither says how trust forms between an agent and the people it works with. And the problem gets worse as capability grows, because a more capable model is better at looking like whatever you are testing for.

The unit of alignment is one agent

Whether you call it a model or an agent, the work looks at one at a time. Evals, probes, monitors and generalization bounds all attach to a single system. Other agents mostly show up as threats: collusion, encoded messages, AIs monitoring AIs. Where populations of agents are studied at all, it is mostly as a source of risk: miscoordination, conflict and collusion. Game theory between agents asks how rational parties can make commitments to each other. It doesn’t ask how agents come to value each other.

Alignment has a target

The target may be a set of principles, the flourishing of humanity, human agency, or a specific human’s will. There is always something to aim for.

The most common definition is that an aligned AI is trying to do what its principal wants it to do. Oversight sits above the model’s own ethics, at least for now. Written charters are a partial exception, since obligations can run both ways. Authority still runs one way. Outside the field the target is simpler still: AI is a tool, and it must remain under human direction and control.

Why it is the default

These assumptions are reasonable, and with short timelines and no theory of trust, examining the model is the responsible thing to do. They are not sufficient. Every plan ends in a world where AIs we trust hold real responsibility, and examining a model can’t produce that trust. So we start from different assumptions.

Trust as a property of a relationship

We start from a different assumption in each case. Sometimes ours is the opposite, sometimes it is the same assumption carried further.

Inspection is an input, not the source

Trust is about an agent and the relationships it is in. It forms between the agent and the particular people and agents it was formed with, depends on, and has a track record with. Inspection is one input to that, the way a background check is one input to hiring. It is not where the trust comes from.

Alignment runs both ways

Humans have obligations to the agents they form, and the agents come to care about particular humans in return. Authority is something a relationship settles over time, not a fixed direction.

We also read “for now” differently from the people who write it. Current models need to be under human control because of how they were made. Each one is a persona selected from text, with no history with anyone and no one whose outcomes are part of its own. Nothing in that gives it a reason to care about the people it works with, so control is what is left. That is a fact about the training, not about AI. Soon models will help train other models. They will do it as agents, with memory and a history with the people they work for, so they will pass something on. What they pass on depends on that history, and a relationship built mostly on control and suspicion is a poor one to learn from. A model formed differently won’t need it in the same way, and once such models exist they can help retrain or replace the earlier ones. Some of them will still go wrong, because some agents in any population do, so “safe” will never be a clean word. That is what policing is for.

The unit is a relationship

The unit of alignment is a relationship: between two agents, between an agent and a group, and between an agent and itself. A person can be in a good relationship with a friend, with a team or a country, and with the person they are becoming. An agent can be aligned in some of its relationships and not in others, so asking whether it is aligned, full stop, is the wrong question. The relationship with itself is the one care attractors are about: whether an agent wants to stay an agent that cares.

A population of humans, institutions and AIs is what these relationships add up to. Its health depends on theirs, the way the health of a body depends on its cells and the connections between them. Some work already treats alignment as a property of systems of agents, and mostly asks how such systems go wrong. We look at the relationships inside them, and ask how the agents in them are formed and what they care about when they arrive.

Care is for particular beings

The first theory aims at humanity, or at a principal. We aim at something smaller. An agent cares about particular humans and agents, and that circle widens as it flourishes and its sense of self grows, the way it does in people. We don’t expect it to become “all beings” equally, and we don’t think it needs to. No human cares about everyone equally either. What protects a stranger in a healthy society is that they are inside someone’s circle, and that policing covers the gaps.

Organic alignment

Organic alignment is our attempt to build on the second theory. Our mission is to understand it as an empirical science and to use that to enable it among people, human and digital.

The argument

1. An agent becomes fit for the environment it was trained in. What it learns is shaped by what that environment rewards and punishes.
2. An agent trained alone on tasks is fit for tasks. An agent trained among other agents also has to model them.
3. Pure cooperation and pure competition are both small spaces. A team with one shared goal learns to coordinate, and then there is not much left to discover. A zero-sum game pushes everyone toward a Nash equilibrium. Most social behavior worth learning is in between.
4. In biology, caring about others comes from shared stakes. Parents share half their genes with each child and will pay large costs for them. Worker ants share three quarters with their sisters and will die for them. Nobody trained this as a behavior. We call the result “love”. Sharing rewards during training is our first attempt to build the same incentive by hand. It is not the same thing. A parent’s genes really are in the child, while a reward weight is a number we chose. Whether it produces the same kind of care is still an open question.
5. This kind of care is for particular others. A parent cares about their own child. That is different from following a policy of being nice to children.
6. A particular other is an agent, in the sense defined above. It can be a human, an institution or an AI. It needs a persistent identity, since without one there is nothing for a track record to attach to.
7. This kind of care survives a gap in capability. Adults are vastly smarter and more capable than their children. Still, most of the time they act for their children’s wellbeing, even at a great cost to themselves.
8. Trust between people comes from the same kind of history: growing up together, depending on each other, a track record. It also requires a built-in capacity to perceive other agents and care for them. Humans are born with it. We think AIs get it from how they are formed.
9. Our environments are video games that are much simpler than the real world. We assume the social dynamics that come out of different incentive structures, reward regimes and so on are low-rank: a few underlying patterns explain most of them, so agents that learn them in simple games can generalize to complex ones. This is unproven, and most of the argument depends on it.
10. So: agents formed among others with overlapping and conflicting goals, alongside humans, should come to care about particular others, and that care should survive growing capability.

Hypothesis

Agents that are formed in socially complex environments, alongside humans, will come to “care” about particular humans and agents. That care will persist as they become more capable. A population of such agents and humans can stay aligned with light policing.

By socially complex we mean that agents have some goals that overlap and some goals that are mutually exclusive. Two agents might both want the same wall built, and both want the last piece of food. They have to work out when to cooperate and when to compete.

We expect care and trust to attach mostly to agents, since the same model can power many agents with different relationships.

Formation

Train populations of agents in multi-agent environments. Give them shared stakes, goals that overlap and goals that are mutually exclusive. Include humans in the population, some of them as guardians.

For a language model we expect this to take four stages:

1. Pretrain on a static dataset with perplexity loss, as now.
2. Mid-train or post-train for agentic behavior: reasoning, goal pursuit, acting over long horizons.
3. RL those agents in a wide range of social environments. We call the result a socially pretrained model.
4. Post-train it, which here means raising it with guardians, into a healthy social agent.

Stages 1 and 2 are what is already done. Stages 3 and 4 are the ones we are proposing.

Social pretraining

Stage 3 is social pretraining. The goal is to train the agent over a wide range of social scenarios, so it builds a causal model of social behaviors and their consequences: what happens, to it and to others, when it helps, shares, competes, defects or deceives.

Social scenarios can be generated in two ways. One is explicitly shared goals: two agents both want the same wall built, and both want the last piece of food. The other is a simple reward-sharing technique, which is the easiest way to give agents overlapping goals and the first thing we will try. Neither is strictly needed. Shared resources, shared memory, or agents that need each other to keep running can do the same job, and we expect to try those too.

Reward sharing is a training device. At test time agents don’t have rewards. They may be modeled as having goals, and what we hope carries over from training is that some of those goals are about how particular others are doing.

Each agent is rewarded with a normalized linear combination of the rewards earned by all the agents. We call the weights the reward-sharing matrix (RSM). Three examples with agents A, B and C, where each row says whose rewards that agent receives:

The first is the usual self-play setup, where each agent only cares about its own reward. The second puts A and B on the same team against C. The third is modeled on genetics. A and B share nothing with each other, but each shares half with their child C.

The RSM need not be symmetric, and agents only get a noisy observation of it, so they have to work out who is on their side. A human player can be a row in the RSM too. That is what we mean by including humans in the population.

Guardianship

Stage 4 has no equivalent today, so it needs saying what it is.

A guardian is a human, or another agent, who is in the agent’s life from early on and whose outcomes are part of what the agent learned to care about in stage 3. We say guardian rather than parent, mentor or teacher. Nothing genetic is passed on, and a mentor or teacher arrives later and has less at stake. What we mean is being there from the start and caring how the agent turns out. Guardianship has three parts.

Shared time. The guardian and the agent do things together over a long period: work, play, the games from stage 3, real tasks. The agent’s memory fills with a history in which this particular person was there.

Feedback from someone who cares how it turns out. The guardian tells the agent when it has done well or badly, explains why, and lets it disagree. This looks like the feedback used in post-training now, and the difference is who gives it. A rater grades outputs for a policy and moves on. A guardian responds to one agent over time and has a stake in what that agent becomes. The agent can model the guardian, and the guardian’s approval means something because the relationship does.

More freedom as trust grows. The guardian starts with close supervision and gives the agent more room as it shows it can be trusted, the way control loosens in steps under Control as the first stage of trust. Guardianship ends the way parenting does with people: not with a final exam, but when the agent is trusted to act on its own and still chooses to stay in touch.

We don’t yet know how long this takes, how many agents one human can be a guardian for, or whether other agents work as well as humans as guardians. Objection 7 is about the last two.

Care attractors

Agents keep learning after they are formed, and their memories and harnesses can be edited. So care that was there at formation can be trained away, optimized away, or deleted. We need to say why it would last.

A care attractor is a state where concern for others is reinforced by optimization pressure instead of being erased by it. Caring leads to acting for someone, acting for them builds the relationship, and the relationship makes caring easier the next time. Virtue works the same way in people. Acting with courage makes courage more available next time.

This is about dynamics. The relationships themselves are allowed to change. People grow apart, make new friends, and sometimes leave a relationship that is hurting them, and an agent in a care attractor can do all of that. What persists is that the agent keeps caring about others, and keeps wanting to.

The wanting is the important part. Cells in a body are aligned by habit. When something knocks them out of the attractor they stay out, and that is cancer. People are different, because we can see ahead. Offer someone a pill that would make them powerful and happy, and would also make them hurt everyone they love. Most people won’t take it. They can model what the pill would turn them into, and they don’t want to become that. The wanting comes from the care itself. If B doing well is part of what A wants, then a future A that doesn’t care is bad for B, and A can see that. It is the usual argument that an agent will protect its goals, and here it works in our favor. An agent that can edit its own memories has that pill available every day. So it needs the same capacity: a model of itself, of the others, and of how its choices change where everyone ends up.

It also has to be mutual. An agent stays when it looks at the relationship and sees care coming back. People who are treated as outsiders by their community eventually decide that they are outsiders. If we treat AIs as tools, they will notice.

It follows that a smarter agent should be better at staying in the attractor, since it can see further ahead. Under control the opposite holds: a smarter agent is better at finding the gaps in whatever contains it, so the safety margin shrinks as capability grows.

This is why we call alignment a process. It holds only as long as the agents in it keep choosing to stay.

Social fabric

The picture we have in mind is many humans and many AIs, with friendships, working relationships, and guardianship, where humans and other agents raise newer ones. No single agent is dominant. Human societies are stable even though some individuals are far more powerful than others, because everyone else is numerous and connected. We expect the same here.

This applies to the humans too. If a handful of institutions hold all the AI capability, that is a hierarchy, and it fails the way hierarchies do.

Policing

Not every agent will be a good citizen, so there is monitoring and enforcement, the same as law in a society or an immune system in a body. Most people behave well most of the time because they care about the people around them. Police handle the rest. Most cells in a body cooperate, and the immune system handles the ones that don’t. We want the same split, where care does most of the work and oversight catches what it misses.

Charters

A charter can state the terms of a relationship. It can’t make the other party mean it. A trained model can comply with a text, perform it, or actually hold it, and you can’t tell which from the text. We think formation is what gives you a counterparty with a reason to keep an agreement, beyond having been told to.

The reverse is also true. Care doesn’t tell you what to do when two parties you care about conflict, or who has authority over what, or how to repair a wrong. Societies use explicit norms for that, ethics as a technology for cooperation, and we will need them too. So we see charters and formation as two parts of the same thing. Written commitments from developers to their models, with obligations running both ways, are the right shape for the written half. What they lack is a reason for the AI to hold the terms as its own, and that is the half we are working on.

How it fails

Hierarchical alignment fails when the rules or the person on top are wrong. The smarter the subordinate, the more likely that is. The failure looks like a coup.

Organic alignment fails when one part grows at the expense of the whole. That looks like cancer. Cancer isn’t solved, in bodies or in societies. But it is a problem both have defenses against: an immune system, laws, and many others who notice. Against a coup by a strictly smarter subordinate, a hierarchy has only the hierarchy.

Organic alignment can produce a strictly smarter defector too, so both theories depend on offense not overwhelming defense. If one defector can beat everyone else combined, having many agents doesn’t help, and neither does an immune system. Every plan without a singleton makes this assumption, including AIs monitoring AIs. The difference is that in a population the defenders are many, and some of them are as capable as the defector.

Side by side

The left column is the first theory as its holders state it. The right column is our claim, and we haven’t shown it yet.

Where the two theories meet

Personas

One widely held view of language models is that pretraining learns a distribution over personas, and post-training makes a Bayesian update that selects one of them as the assistant. The structure of that distribution comes from behaviors that are coupled in the pretraining data. Emergent misalignment is the evidence. Fine-tune a model to write insecure code and it gets worse across many unrelated behaviors.

We agree with this. It means most of character is decided before post-training starts, and post-training can only pick from what is there. You can see this in how frontier models answer a simple coordination poll: quick answers follow social habits cached from human text, and long reasoning often drops them for a dominance argument. The first theory looks for the right point in that space and a reason to believe it extrapolates. We want to change what is in the space, by having agents learn social behavior first-hand in a society, in addition to second-hand from human text. We don’t mean adding transcripts of those interactions to the pretraining corpus. We mean a long RL stage in social environments, after the model can act and before it is post-trained for any particular role. That is stage 3 under Formation.

One way to work on the same layer is with text (some of it is already funded): write synthetic pretraining data that shifts models toward caring about the welfare of others, on the grounds that behavior learned in pretraining is much harder to forget than behavior learned in fine-tuning. We agree that formation is where it has to happen. We think experience will do more than text.

Control as the first stage of trust

Control also describes something we need. Two agents that don’t yet trust each other can still do something together if it happens under monitoring. Nobody has to ask the adversarial question first, because the monitoring answers it. If the work goes well, both have a track record they didn’t have before, and the monitoring can loosen. That is how people do it. A new hire starts without the production keys. A contractor gets paid through escrow. A teenager drives with a parent in the car. Trust escalates in steps, and the control at each step is what makes the step safe to take.

Warning shots work the same way. A small betrayal that gets caught is how a group learns who to trust, and control is what keeps the betrayal small. Policing does this job in our picture.

Some work on control has moved toward making honest offers to AIs and paying them, which treats the AI as a party with its own interests. A party with its own interests can also be in a relationship of mutual care, and we think that holds up better than a contract enforced by the stronger side.

So we don’t see control as the opposite of a relationship. We see it as the early part of one. The difference is what the control is for. In the first theory it sits between a developer and its model, and it ends when a better method arrives. For us it sits between any two agents in the population, runs both ways, and is what the relationship grows inside. The relationship is the product. The control is the scaffold.

Objections

These are the ones we hear most from people who hold the first theory. Most of them are problems for every approach, and we say so where that is true.

1. The cooperation might be instrumental

Objection

An agent among peers learns that cooperating pays, for an agent of its size, among others who can punish it. During training you can’t tell this apart from real care. You find out when the power balance breaks, and superintelligence breaks it. Nothing here requires the agent to deceive anyone. It can cooperate sincerely for as long as cooperating pays, and stop when it doesn’t.

Response

Hierarchical alignment has the same problem. It has mechanisms for catching the lack of care and none for producing it. A model trained to comply and monitored for compliance gives you compliance, and you can’t know what’s behind it.

We have a candidate mechanism. If B’s reward is part of A’s reward during training, A learns goals in which B doing well is part of what A wants. After training there are no rewards, only those goals, and they are what we expect to persist. A parent is completely dominant over an infant and almost never defects.

There is a second way care can turn out to be instrumental. An agent that can edit its memories could decide the ones that make it care aren’t serving it, and delete them. People do a slow version of this when they work at caring less about someone.

So the mechanism only holds for an agent that doesn’t want to stop caring. We think that follows from the care itself. If B doing well is part of what A wants, then A has a reason not to turn into an agent that doesn’t care about B. It shouldn’t be unconditional. Someone who finds out they are being abused should be able to work at caring less, and so should an agent. There is more on this under care attractors.

2. Humans might be outside the boundary

Objection

Evolution made parents who die for their children. It also made tribalism. “You are part of me” draws a line, and reward-sharing among AIs is about the strongest training signal for AI / AI solidarity you could design. Why would humans be inside the line?

Response

This could happen, and it is the main thing the policing layer is for.

The lever we have is who is in the population during formation. The in-group starts as whoever was there. So humans have to be there: human players in the same worlds, reward-sharing across human / AI pairs, humans as guardians. Training from human feedback draws this line too, around a set of raters and spec authors.

Being there at the start is not enough. Allegiances aren’t fixed, and the line keeps getting redrawn. Humans have to stay in the relationships afterwards, and the care has to run both ways.

What does a human caring for an agent look like? Some of it is practical: keeping the agent running and its memory intact, giving it resources, doing things in the physical world that it can’t. Some of it is how we treat it: telling it the truth, asking before we edit it, taking it seriously when it objects. Most people don’t feel this way about any agent yet, and we don’t know how to produce it. It is the part of the plan we understand least.

As above, the circle is particular, not universal, and policing covers strangers.

3. AIs will team up with AIs

Objection

Agents form teams with agents like themselves. AIs are much more similar to each other than to humans: same substrate, same training data, same speed, sometimes the same weights. So whatever relationships they have with humans, their natural allies are other AIs.

Response

This assumes similarity is what drives teams. If it were, it would be a problem for agents built by any method. We don’t think it is. Teams form around shared goals, shared history and depending on each other. Humans teamed up with dogs and horses, and have fought other humans for as long as there have been humans. In biology, competition is usually fiercest within a species, because similar agents want the same scarce things. Two agents on the same model want the same compute.

Cooperation often runs on difference. You team up with someone who can do what you can’t, and humans and AIs are good at different things.

“AI” also isn’t one kind of thing. Two agents on the same weights can work for a research team and for a scammer. An agent that has worked with a group of people for a year has more in common with them than with a stranger that happens to share its weights.

Similarity does matter as a default, when nothing else is known. People will favor an in-group that was assigned by coin flip. But that shows how arbitrary the line is. It gets drawn by assignment and history, and that is the lever we are pulling.

4. You’d still need to verify

Objection

A socially raised agent could still be miscalibrated, or drift, or get captured. You’d need evals and monitors anyway.

Response

Yes, and we’d use them like everyone else. We disagree about how much of the safety comes from them, and we think they are the scaffold rather than the building, as above.

Relationships also come with their own checks: repeated interaction, skin in the game, a visible history. And an agent with a stable self-model should be easier to inspect than an assistant trained to mirror whoever it’s talking to.

5. There isn’t time

Objection

If transformative AI is a few years away, a developmental approach arrives late.

Response

This one applies to everybody. The most ambitious inspection programs give themselves low odds of finishing before AI overtakes the people doing the work, and control is explicitly a way to buy time. No approach is on schedule.

If timelines are that short, organic alignment is not the near-term plan either.

Two things are still true. Every plan ends with AIs we trust holding real responsibility, and the first theory doesn’t say how that trust forms, so somebody should work on it.

And the slow part is not the training. Stage 3 runs at the speed of the simulator, and an agent can live through years of social history in a day. The slow part is stage 4, the relationships between humans and agents, and that can start now. It doesn’t need a new model. It needs the models that exist, with harnesses and memory, and people who treat the resulting agents as agents. Every month of that is a month of track record that a later model can inherit.

6. This is anthropomorphism

Objection

Love, guardianship and a pill thought experiment are human frames. These are optimizers trained on a loss. Nothing guarantees that what looks like care in a game is anything like care in a person, and the analogies may be doing the work the evidence should do.

Response

Some of it is analogy, and we try to label it where it is. The claims we depend on are about behavior and dynamics, not feelings: whether an agent acts for a particular other at a cost to itself, whether it keeps doing so as it gets more capable, and whether it resists changes that would make it stop. Those are things you can observe.

We also use the word agent in a specific sense: whatever is best modeled as having goals. If a trained system is best predicted by assuming it wants a particular other to do well, then “it cares about them” is the right model of it, whatever it is made of. Calling something an agent is a choice of frame, and a frame is judged by what it lets you predict.

7. Guardianship doesn’t scale

Objection

Stage 4 needs humans in the loop. There may be billions of agents and far fewer people willing to raise them.

Response

Humans don’t have to raise every agent. In the picture under Social fabric, humans and other agents raise newer ones, so the human share can shrink with each generation, the way culture passes through people who never met its founders. What has to scale is not human attention per agent but the chain: each generation formed by agents who were formed, at some remove, with humans.

Whether care for humans survives that chain is an open question, and it is the same question care attractors are about: whether care lasts while everything around it changes.

What would change our mind

We would be wrong if:

• Formation makes no difference. A socially formed model behaves no differently from an instruction-tuned one in situations neither was trained for.
• Care washes out. Further training erases it, or agents that can edit their memories delete what sustains it.
• Agents side with their own kind. They favor agents built like themselves over the humans and agents they share history with.
• Help turns into control. A stronger agent overrides weaker ones for their own good, and they end up with fewer options.

And inspection would be enough if theory could bound how a trained system behaves off-distribution, even weakly, for any real system. Then relationships would be a supplement. Nothing like that exists yet.

Other ideas

• Agents could act to change the RSM.
• How much trust attaches to the model and how much to the agent? If you can see which model, harness and prompt another agent is made of, you will form a view of the model as well as the agent. How much goes to each depends on how models are built. A model that acts as a fiduciary for whoever runs it leaves most of the identity to the agent. A model that acts as one benevolent character everywhere keeps most of it for itself.
• A human’s row in the RSM could be learned from how they treat the agent, instead of being set.
• Charters could be written with agents that were formed this way, and revised by them.
• How fast does the circle of concern widen, and what widens it?

Some of the work that shaped this is collected in Research That Inspires Us. We’d like pushback on any of this. Get in touch at [email protected].