The Artificial Experimentalist Discovery and Control of Self-Organizing Phenomena with Autotelic Reinforcement Learning
We introduce CARL, a reinforcement learning agent that discovers and controls self-organizing patterns in Lenia, a continuous cellular automaton. Trained agents can (1) create self-organizing patterns, (2) control the patterns' behavior, and (3) follow human directional commands in real time to guide patterns through a maze, as shown in the three clips below, from left to right. All three run on the same loop, shown in the diagram underneath them: CARL samples a goal at the start of an episode, then at every step locally adds or removes cell activations, after which the Lenia update rule advances the state.
Try it yourself in the browser: Create a soliton → Steer a soliton →
Abstract
Existing methods for exploring cellular automata and other complex systems mostly operate in open loop: they set initial conditions, execute a full simulation, and observe the outcome, without intervening during execution. We introduce a closed-loop framework based on autotelic reinforcement learning, in which an agent autonomously samples diverse goals and learns a goal-conditioned policy to intervene in a complex system through minimal, local perturbations. We instantiate this framework on Lenia, a continuous cellular automaton known for life-like self-organizing patterns, in an agentic system we call CARL, and demonstrate three capabilities. First, CARL discovers stable solitons across a wide range of Lenia update rules at a higher rate than heuristic baselines. Second, it learns to steer the movement direction of existing solitons with few interventions, showing that CARL can control self-organizing patterns, not only create them. Third, humans can use trained agents to guide solitons through maze environments in real time by specifying high-level directional commands that the agents translate into low-level interventions. Trained across diverse goals, update rules, and random initial states, the agents acquire policies that generalize zero-shot to various out-of-distribution conditions. These results suggest a path toward artificial experimentalist agents that, autonomously or with human guidance, discover and control emergent phenomena in complex systems.
Introduction
One of the central endeavors of science is understanding complex systems at all scales of organization, from elementary physics to astronomy, from molecular biology to ecology. Two general goals drive this research: (1) explaining and discovering the diverse phenomena that emerge in complex systems, and (2) learning to control complex systems toward desired states, ideally with minimal effort.
In biomedicine, for instance, the goal is not continuous intervention, but the restoration of healthy, self-sustaining dynamics. Rather than controlling individual components, we aim to guide systems back into stable regimes in which they can maintain their function autonomously. However, identifying such interventions is inherently challenging, as system behavior arises from interactions across many scales. This raises a fundamental question: how can we systematically control systems whose internal dynamics are complex or only partially understood (Levin, 2023, 2025)?
Computational approaches are essential for pursuing these goals, as they enable us to simulate complex systems in silico. Cellular automata (CAs) have long served as standard models for studying self-organization, and recent continuous extensions such as Lenia (Chan, 2020) produce increasingly complex and life-like patterns, making them ideal testbeds for discovering and controlling emergent phenomena. Because the space of behaviors a CA can produce is vast and difficult to anticipate, researchers have developed a range of methods to explore it (see Related Work in the paper). Most of these, however, operate in open loop: parameters and initial conditions are chosen upfront, with no interaction during rollout.
This stands in contrast to how humans typically engage with complex systems: continually observing and interacting with the system in real time to form an intuition of its causal dynamics (e.g., a gardener continuously pruning, watering, and reshaping a garden as it grows). As systems grow in complexity, however, effective intervention becomes increasingly difficult. Nonlinear interactions, feedback loops, and delayed effects make human intuition prone to systematic biases, and modern challenges — particularly in biomedical, ecological, or economic contexts — quickly reach the limit of human modeling capabilities (Tversky and Kahneman, 1974).
To address these gaps, we propose an autotelic reinforcement learning (RL) framework as an interaction-driven approach to translate a diversity of desired experimental outcomes into actionable, causally effective interventions in complex systems. By autotelic, we mean an RL agent that autonomously self-generates and learns to achieve diverse goals — the operational counterpart of these outcomes — in the considered complex system (Colas et al., 2022). As a concrete instantiation, we introduce CARL (for controlling Cellular Automata with Reinforcement Learning), which learns to intervene on CAs in a closed-loop fashion, continuously observing and shaping their dynamics over time (Fig. 1).
Through a series of experiments using Lenia as a testbed, we demonstrate that CARL can both discover interesting local self-organizing phenomena, called solitons, and control their behavior, with limited interventions and in a sample-efficient manner. Moreover, CARL generalizes well to out-of-distribution scenarios, such as unseen world dynamics and action spaces. Lastly, we show that trained CARL agents can be deployed interactively, enabling humans to specify and adjust high-level goals in real time while the agents translate them into low-level interventions for controlling complex systems.
While the framework is demonstrated on Lenia, it is designed to be system-agnostic. Looking ahead, we hope to extend it to increasingly biologically grounded models.
A motivating example
How Lenia works
A Lenia world is a grid of values in [0, 1] with periodic boundaries. Every cell sums up its neighbourhood through a fixed kernel K, and a growth function φ turns that sum into a change applied to the cell:
with the result clipped back into [0, 1], and dt = 0.1 the step size throughout. The same rule applies to every cell at once.
The kernel is a disk of radius ρ, divided into b concentric rings of equal width. Each ring carries a smooth bump scaled by its peak value βi, and the whole kernel is normalized to sum to one. So ρ and β = (β1, …, βb) set what each cell perceives.
The growth function maps the kernel sum into [−1, 1] through a Gaussian bump:
A cell grows when its neighbourhood sum lands near μ and decays as the sum falls away from it; σ sets how wide that window of tolerance is.
Lenia's dynamics are sensitive. An orbium, the best-known Lenia soliton, glides across the grid indefinitely when left alone, and a patch of randomly raised and lowered activations dissipates within a few steps on an empty grid. What happens when we combine them?
The outcome is not the same each time. Under different random draws, the same kind of patch can rotate the orbium, dissolve it, blow it up into a Turing-like pattern that fills the whole grid, or cause it to morph into an entirely new stable soliton. A few perturbations at the right place and moment decide which attractor the system falls into. In this work we ask, can we train a controller that knows where and when to perturb, in order to steer the system towards desired regimes? More specifically, can we train a controller that can discover new self-organizing patterns, and learn to control their behavior?
Framework
We cast the discovery and control of solitons in Lenia as a sequential decision problem, and tackle it with reinforcement learning (Fig. 1).
An episode alternates between the agent and the system. At each step the agent observes the grid and applies one intervention, the Lenia update rule then advances the state, and the agent receives a reward that scores the new state against the goal and charges for the intervention. Interventions are local and simple: an action adds or removes mass in a small disk around a single cell. The agent may also choose not to intervene.
Three things are sampled at the start of each episode: the goal, the target the agent must reach; the action cost, the negative rewards incurred for every intervention; and the update rule, the dynamics the episode runs under. Lenia is essentially a large family of update rules, and we train across a set of rules rather than a single one.
The action cost shapes how the agent reaches its goal. The agent can intervene at every step, micromanaging the system, or place a few interventions well and let the system's own dynamics carry the state the rest of the way. A higher action cost tips the balance toward the latter, pushing the agent to act sparsely and to work alongside those dynamics rather than against them.
The policy network is a U-Net (Ronneberger et al., 2015) that maps the grid to a dense field of Q-values, one per action type. Dense action representations of this kind have previously been used for robotic pushing, grasping and mobile manipulation (Zeng et al., 2018; Wu et al., 2020). Like CAs, the network is convolutional and local; its receptive field is nonetheless wide enough to cover most of the grid. It observes the last four states, so that the dynamics of the system are visible. Goal, action cost, step index and the parameters of the running update rule enter through FiLM conditioning (Perez et al., 2018), so a single policy of roughly 800K parameters trains on many goals and update rules. We train with Double DQN (van Hasselt et al., 2016).
Experiments
We demonstrate CARL through three experiments.
- Soliton creation. CARL creates stable solitons across a wide range of Lenia update rules and from procedurally generated initial states, and trained agents generalize zero-shot to unseen goals, update rules and modified action spaces.
- Soliton steering. A second agent steers the movement direction of existing solitons, demonstrating that CARL can not only create self-organizing patterns, but also control them.
- Human-in-the-loop. A proof of concept experiment that humans can interact with complex systems through CARL agents by modifying the agents' goals in real time, by having users navigate solitons through a maze using the soliton steering agent.
1 · Creating solitons
Create a soliton yourself →The phenomena of primary interest in Lenia are solitons: spatially localized, self-organizing patterns that persist over time and often move across the grid. Solitons are not easy to find; most initial states and update rules result either in dead worlds or global, Turing-like patterns that fill the whole grid. A range of search methods have been developed for the purpose of discovering them, spanning from systematic exploration of the update-rule parameter space, paired with hand-designed initializations of the grid (Chan, 2020; Hudcová et al., 2025) , to population-based diversity search methods that accumulate an archive of distinct behaviors, such as intrinsically motivated goal exploration processes (Etcheverry et al., 2020) and quality-diversity (Faldor and Cully, 2024).
How we detect solitons
To detect whether a soliton is present on the grid, we use a simple filter inspired by prior work (Hamon et al., 2025; Faldor and Cully, 2024): after applying the Lenia update rule for 5,000 steps without intervention, we classify the resulting state as containing a soliton if its total mass is non-zero and below 10% of the grid capacity. We omit additional checks used in prior work — such as a velocity threshold, mass variability, and robustness testing — since our goal is to evaluate CARL for its ability to create local, persistent patterns of any kind. We visually inspected many simulations that pass the filter and found no false positives (i.e., global, Turing-like patterns were never accepted).
Rather than explicitly training to build solitons — which would require defining what constitutes a soliton within the reward signal — we train CARL on a simpler, proxy task: maintaining a target mass on the Lenia grid, for a given update rule and action cost. When actions are costly, the agent faces a choice between constantly intervening to hold the mass at the target, and finding a self-sustaining configuration that matches it. Here the second option is simply cheaper: a configuration that holds the target mass without help costs nothing to maintain. Soliton creation therefore emerges as a byproduct of reward maximization.
Goal space and reward, more formally
The goal space is 𝒢 = 𝒯 × 𝒞 × Ω, where 𝒯 is the set of target masses, 𝒞 the set of action costs, and Ω the set of update rules. At the start of each episode, CARL uniformly samples a goal g = (τ, c, ω), with τ ∈ [0, 200], c ∈ [0, 0.2], and ω drawn from a set of 85 training update rules: soliton-supporting rules hand-selected from the parameter-space survey of Hudcová et al., 2025, all at kernel radius ρ = 10. The reward at each step is
where Mt = ∑i,j Xti,j is the total mass of the grid at time t, N the total number of grid cells, and δt ∈ {−1, 0, +1} the action type selected by the agent. The first term penalizes deviation from the target mass, the second penalizes non-trivial interventions, weighted by the sampled action cost c. A single policy must therefore learn to act across diverse combinations of target masses, action costs, update rules, and initial conditions. Sampling the action cost per episode rather than fixing it serves two purposes: it produces a single budget-adaptive policy that can operate across a spectrum of intervention regimes at deployment, and we speculate it also acts as an exploration mechanism during training — low-cost episodes allow the agent to freely discover viable configurations, while high-cost episodes pressure it to find self-sustaining ones.
Which update rules the agent trains on
t = 0
t = 1
t = 7
t = 21
t = 101
t = 521Results
We evaluate the trained agent on the Cartesian product of target masses τ ∈ {0, 25, …, 400}, action costs c ∈ {0, 0.05, …, 0.4}, and all 85 training update rules, running 16 episodes per combination. Both τ and c extend to twice their training range.
Comparison with baselines
We compare CARL against several heuristic baselines across all training update rules, with a fixed action cost of c = 0.1 and mass targets τ ∈ {50, 100, 150, 200}.
- No-op. Never acts; measures how often the initial conditions alone produce solitons.
- Random. Random action type and location.
- Mass (near). Adds or removes mass toward the target, at cells that already hold mass.
- Mass (rand). The same rule, but at a uniformly random location.
- Mass+thr (near). Mass (near), acting only when the board mass deviates by more than 10% from the target.
- Mass+thr (rand). Mass (rand) under that same 10% threshold.
Generalization. The results above show that CARL reliably creates solitons under training update rules. We now assess how robust this capability is along three axes:
- Modified action parameters. Can the agent still act effectively when its own interventions change in reach and in strength, while the world stays the same?
- Rescaled update rule kernels. Can it adapt to update rules from the training set when they are stretched to a different spatial scale?
- Entirely novel update rules. Can it handle rules it never trained on, which support an entirely different repertoire of patterns?
Modified action parameters
At every step the agent picks a spot on the grid and adds mass, removes mass, or does nothing. Two parameters set what that means: the action radius Ra, how far the intervention reaches, and the action magnitude Ma, how much every cell within that radius moves up or down, clipped to stay in [0, 1]. Both are fixed during training, at Ra = 5 and Ma = 0.3, so all the agent chooses is where to act and which of the three actions to take.
Here we change Ra and Ma at test time, with τ = 125 and c = 0.1 fixed. Note that the agent does not observe the action hyperparameters.
Rescaled update rule kernels
We deploy the agent across a range of kernel radii ρ ∈ {4, 6, 8, …, 26}, proportionally adjusting the action radius and grid size (linearly) and the target masses (quadratically), so that the ratio between action scale, target mass and available space stays roughly consistent. Since the policy network is fully convolutional, it can be deployed on different grid sizes without modification.
Note that this is a particularly challenging form of generalization: rescaling the kernel does not simply scale the emerging patterns, but can fundamentally alter their behavior due to discretization effects. The same update rule at different kernel radii can produce solitons with different shapes, sizes, movement patterns, and levels of robustness.
This experiment also carries a practical benefit. Experiments with large radius ρ are highly expensive, so training at a small radius and deploying at a large one amounts to fitting a controller in a cheap simulator and running it at high fidelity.
Entirely novel update rules
Finally, we deploy CARL on unseen convolutional kernels. We select seven kernels K not included in the training set and sweep across growth function parameters μ ∈ {0.2, 0.205, …, 0.4} and σ ∈ {0.02, 0.022, …, 0.06}, with τ ∈ {50, 100, 150, 200, 250} and c = 0.1, at kernel radii ρ ∈ {10, 14, 18}. A trained agent can map new update rule spaces for regions supporting solitons, in a sample efficient manner. We show three of the seven kernels below.
2 · Steering solitons
Steer a soliton yourself →To demonstrate that CARL can control self-organizing phenomena, not only create them, we train a new agent on a task where it must steer a soliton toward a target direction. Each episode begins with a uniformly sampled target direction, an action cost as before, and a soliton drawn from a set of 48 that the first agent discovered — placed on the grid under its own update rule and randomly rotated.
A soliton's movement direction is measured by its center-of-mass displacement over the last four timesteps. The reward is the sum of three terms: 1) the cosine similarity between this direction and the target direction, which rewards alignment; 2) a mass penalty, as in the previous task, but with the target mass set to that of the initial state; and 3) the action cost, as before.
Goal space and reward, more formally
The goal space is 𝒢 = 𝒟 × 𝒞 × 𝒮, where 𝒟 is the unit circle of target directions, 𝒞 the set of action costs, and 𝒮 the 48 solitons the creation agent discovered in the kernel-scaling test at radius ρ = 18 — the larger radius is used because solitons at smaller radii tend to move chaotically. At the start of each 200-step episode CARL uniformly samples a goal g = (d, c, s), with c ∈ [0, 5] and the soliton s placed on a 115×115 grid. The reward at each step is
where vt is the soliton's center-of-mass displacement over the last four timesteps, so cos(vt, d) scores how well its current heading aligns with the target direction. The remaining two terms are mass deviation and the action cost, as before. Here, the mass term acts as an auxiliary loss: its target is the mass of the initial state, penalizing the agent when the soliton is changed too strongly.
t = 2
t = 52
t = 53
t = 62
t = 82
t = 127This is the same effect action costs produced in the creation task, in a different guise. Rather than continuously correcting the soliton's trajectory, the agent learns to apply a brief perturbation that redirects it and then withdraws, letting the soliton carry on along the new heading unassisted.
Generalization. To test whether the policy transfers beyond what it was trained on, we hold out both solitons and directions: the 48 solitons are split into 24 training and 24 holdout, and the circle of target directions into four quadrants, of which only two opposite ones are used in training. Each of the four resulting conditions is evaluated over 128 episodes, averaging the cosine similarity across all 200 steps of each. Performance on unseen solitons is essentially unchanged — 0.91 ± 0.10 mean cosine similarity against 0.91 ± 0.14 in the training condition — suggesting the policy captures direction-dependent strategies that transfer across soliton morphologies. The drop for unseen directions (0.81 ± 0.14, and 0.76 ± 0.15 when both are held out) indicates that the mapping from direction to intervention pattern is only partially learned.
3 · A human in the loop
Drive a soliton through the maze yourself →We demonstrate how trained CARL agents can serve as real-time interfaces for human control. We extend the steering environment with procedurally generated mazes, where walls are regions in which grid values are fixed to zero. A soliton is placed in the maze, and the user can modify the agent's goal (the target direction) and the action cost in real time, steering the soliton through the maze.
The agent has no explicit representation of the maze — it perceives only the single-channel Lenia grid, identical to its training setting. Furthermore, the agent was never trained with changing goals, yet it successfully redirects solitons multiple times within a single episode while preserving their coherent shape. Reducing the action cost makes the agent intervene more frequently and advance the soliton faster, but this tends to be invasive and breaks the soliton's identity.
Although the agent performs well, several failure modes emerge. Wall collisions can cause the soliton to disintegrate or explode, though some solitons are robust to contact. Solitons that move regularly in a straight line tend to be easier to control, while static ones and those moving chaotically need to be constantly micromanaged in order to steer them towards the goal. High action costs can also lead to failure, as interventions become too sparse to maintain the soliton's shape after perturbations, and the agent generally cannot recover a disrupted pattern.
Discussion
We introduce a closed-loop framework for autonomous discovery and control of self-organizing phenomena, based on autotelic RL. Rather than setting initial conditions and passively observing outcomes, a goal-conditioned policy observes the evolving complex system and applies minimal, local perturbations toward diverse self-generated goals. We instantiate the framework on Lenia as a system named CARL, which discovers solitons across a wide range of update rules and procedurally generated initial states, generalizes to out-of-distribution conditions, and can efficiently map novel update rule spaces to identify regions that support solitons. Beyond discovery, CARL agents can also learn to control solitons by steering their movement direction. Finally, we demonstrate that CARL agents can serve as real-time interfaces, enabling human users to guide solitons through maze environments with simple directional commands.
CARL generalizes well to out-of-distribution conditions across variations in goals, action spaces, and novel update rules. This suggests the policies capture transferable system dynamics rather than overfitting. As a result, trained policies can be reused and composed to solve tasks beyond their original training objective. We demonstrate this in the maze experiment, where a human user modifies the agent’s goal (desired movement direction), while the policy handles the low-level control to achieve it in real time. This compositional reuse points toward functional integration, where distinct capabilities can be combined to solve increasingly complex tasks. Such integration suggests a path toward hierarchical control, where higher-level agents or processes set subgoals for lower-level controllers. We see CARL as a step toward artificial experimentalist frameworks, where agents not only learn how to autonomously act on complex systems, but also how to structure and combine those actions — deciding what to investigate through self-generated goals and how to achieve it.
A key limitation is that instantiating the framework requires domain expertise: the reward function, action space, and observation design all encode knowledge about what makes a given system interesting. While the soliton creation task sidestepped the need to define solitons explicitly, it still reflects a designer’s intuition about Lenia. Domain expertise is inherent to scientific inquiry, but when the goal is to uncover phenomena we cannot yet characterize or anticipate, more open-ended approaches—such as intrinsic reward signals, adaptive goal sampling policies, or automated environment and task design—could reduce this dependence and broaden the scope of discovery.
Lenia offers favorable conditions for closed-loop control: full observability, determinism, and a simple action space. The applications we ultimately want to reach offer none of these. In biomedical and bioengineering systems, where guiding self-organizing dynamics with minimal intervention would be most valuable, agents will have to cope with partial observability, stochasticity, and high-dimensional, multiscale dynamics.
Acknowledgements
We thank Barbora Hudcová for insightful discussions and guidance in navigating the Lenia Explorer dataset. We thank members of the Flowers AI and CogSci Lab and the Levin Lab for helpful discussions.
We gratefully acknowledge support for this work provided through a sponsored research agreement with Astonishing Labs and from the Templeton World Charity Foundation, Inc. (Grant ID: TWCF-2021-20606). This work benefited from the use of the Jean Zay supercomputer associated with the Genci grant A0091011996. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the funding agencies.
The authors used generative AI tools to assist with writing the manuscript, developing experiment code and this companion website. The authors take full responsibility for the content.
Citation
@inproceedings{cvjetko2026carl,
title = {The Artificial Experimentalist: Discovery and Control of
Self-Organizing Phenomena with Autotelic Reinforcement Learning},
author = {Cvjetko, Marko and Hartl, Benedikt and Levin, Michael and
Moulin-Frier, Clément and Oudeyer, Pierre-Yves},
booktitle = {ALIFE 2026: Proceedings of the 2026 Artificial Life Conference},
address = {Toronto, Canada},
publisher = {MIT Press},
pages = {45},
year = {2026},
doi = {10.1162/ISAL.a.971}
}
Code and data: github.com/markocvjetko/carl-release. Paper: ALife 2026 proceedings. Corresponding author: marko.cvjetko@inria.fr.