Our ideas about organic alignment, coherence, and open-ended learning systems and how to go about building them have been built on a foundation of research from many sources. We share a selection of those sources below. We’ve bolded some of the ones that seemed most important to us.

Some Lighter Fare

Most of the research on the page is academic work, of the serious and sober variety. We believe this kind of work is important as it is the foundation of most of our own research. However, we also think that sometimes it’s more important to convey ideas in lighter and more accessible ways.

• Songs of Life and Mind
• Neurons Gone Wild
• God Hep Us, Let’s Try To Understand Friston On Free Energy
• Nebulosity
• Life Universe (an infinitely zoomable Conway’s Game Of Life)
• Our Universe Evolved
• Meshworks, Hierarchies And Interfaces

On the formation of functional wholes

Larger wholes formed of parts have their own independent existence, one that transcends the mechanics of the parts themselves. Think about our own bodies. Even though we have many different kinds of cells which all act independently, there can be no doubt that to the same degree the cells exist as meaningful agents, so do the humans made of those cells. Ant nests are as real as ants. Species are as real as organisms.

• Technological Approach to Mind Everywhere
• An Ability to Respond Begins with Inner Alignment: How Phase Synchronisation Effects Transitions to Higher Levels of Agency
• An Operational Information Decomposition via Synergistic Disclosure
• What is Intelligence?
• A Theory of Biological Relativity: No Privileged Level of Causation
• Causal Emergence 2.0: Quantifying Emergent Complexity

On multi-agent cooperation

The foundation of agents coming together to form a greater whole starts with agents being able to cooperate with others who they have not yet cohered with. Before agents come into sync enough that there can be complex webs of interdependence, there needs to be basic interdependence through trade of game theory. This is the seed of care that can develop over time into full alignment.

• Evolving Intrinsic Motivations for Altruistic Behavior
• Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning
• Emergent Reciprocity and Team Formation from Randomized Uncertain Social Preferences
• Biology, Buddhism, and AI: Care as the Driver of Intelligence
• Emergent tool use from multi-agent interaction
• Capture the Flag: the emergence of complex cooperative agents
• Dota 2 with Large Scale Deep Reinforcement Learning

On consciousness, physics, and information theory

One of the underlying ideas that our research is based on is that you can model agents as having beliefs at many different scales. Cells have beliefs, organisms have beliefs, tribes and nests have beliefs. There are some very important question here: is there a distinction between being able to usefully model an agent as having beliefs, and the agent having them? What is required to have a sentient agent, and how do sentient agents vary? What are the fundamental thermodynamic limits on learning? And what’s the relationship between beliefs, experience, and matter?

• The Intentional Stance
• Traces of Consciousness
• A Computational Model of Minimal Phenomenal Experience (MPE)
• The free-energy principle: a unified brain theory?
• Making the Thermodynamic Cost of Active Inference Explicit
• Towards a Geometry and Analysis for Bayesian Mechanics
• The Physics of Learning
• The Computational Unconscious: Adaptive Narrative Control, Psychopathology, and Subjective Well-Being
• Empowerment, Free Energy Principle and Maximum Occupancy Principle Compared

On learning in new domains:

Cooperating with other agents and co-aligning with them is a very difficult problem, because it is inherently non-stationary. Because the environment is also made of learning agents, the environment is no longer inductive in any simple way. Actions impact future observations in unpredictable ways dependent on the changing beliefs of the other agents, in a way that does not predictably converge. This makes learning in non-stationary and open-ended environments crucial for alignment.

• Human-Timescale Adaptation in an Open-Ended Task Space (ADA)
• Associative Conditioning in Gene Regulatory Network Models Increases Integrative Causal Emergence
• AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents
• RL-squared: Fast Reinforcement Learning via Slow Reinforcement Learning
• Open-Ended Learning Leads to Generally Capable Agents AGRec
• Open-Endedness is Essential for Artificial Superhuman Intelligence
• Competency in Navigating Arbitrary Spaces as an Invariant for Analyzing Cognition in Diverse Embodiments
• OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code
• Mastering Diverse Domains through World Models (Dreamer V3)
• Enhanced POET: Open-ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions

On expanding the limits of reinforcement learning

Reinforcement learning could also be called inductive learning: we reinforce policies that have worked before in order to do them again in similar situations. In order to succeed in learning in diverse and non-stationary domains, you have to learn how to learn.

• General Intelligence Requires Rethinking Exploration
• Decoupling Representation Learning from Reinforcement Learning
• Complex Behavior from Intrinsic Motivation to Occupy Action-State Path Space
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Deep Hierarchical Planning from Pixels
• AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning
• Scaling Instructable Agents Across Many Simulated Worlds
• SAMBA: safe model-based & active reinforcement learning

On exploration and intrinsic motivation:

Learning how to learn makes exploration problems far more difficult. Instead of learning a strategy for exploration, the agent needs to learn how to explore the space of exploration strategies. This is a far harder requirement.

• Exploration via Elliptical Episodic Bonuses
• Exploration Unbound
• RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning
• ELDEN: Exploration via Local Dependencies
• CAT-SAC: Soft Actor-Critic with Curiosity-Aware Entropy Temperature

On Neural Cellular Automata:

NCAs are a very natural multicellular approach to learning that is very directly inspired by biological organisms. Also we think it’s just really cool.

• Using Neural Cellular Automata to Robustly Control a Cart-pole Agent
• Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programs