Developmental Systems, a Blog of the Flowers Lab

When LLMs discover the world

Thu, 18 Jun 2026 08:00:00 +0000

Targeted audience: AI researchers, AI engineers

Expected reading time: ~10 minutes

Useful links:

1) What is a world model?

What if I told you that a car is speeding toward a cliff?

You probably didn’t just picture the scene—you also imagined what happens next: the car going over the edge and crashing below.

This automatic response relies on an internal simulation—an implicit model of how the world works. Humans continuously build and refine such internal world models [1,2]. These models are at play whenever we imagine, plan, reason, or react to situations.

In that sense, world models are fundamental to human intelligence.

But what about AI systems?

Recently, building world models has gained significant attention. Approaches such as JEPA and its associated architectures [3,4,5], or DINO-WM [6], aim to learn representations that can predict how observations evolve over time. Concretely, given a sensory input (e.g., a video frame), these models are trained to predict what comes next.

This trend is partly motivated by a perceived limitation of current AI systems. Large Language Models (LLMs), despite their impressive capabilities, are often argued to lack a true understanding of the physical world [7,8,9]. This is largely attributed to their training objectives: next-token prediction and human-feedback maximization incentivize fluent, plausible text generation, but do not inherently require—or reward—building an accurate model of how the world works.

However, the idea of world models is not new [10,11].

In the reinforcement learning literature, world models have long been studied under the umbrella of model-based reinforcement learning [10,11,12,13]. In this setting, an agent learns a model of its environment through interaction, and uses it to simulate future outcomes. This allows the agent to select actions by imagining their consequences before acting—or even to learn entirely from imagined experience.

What has changed, then, is not the idea itself but its renewed prominence—and with it, a deeper question: what kind of world model is needed?

2) Building world models

One increasingly popular answer is to learn world models directly from large-scale sensory data—the path advocated by researchers such as Yann LeCun and instantiated by the JEPA-style architectures and DINO-WM mentioned earlier [3,4,5,6]. These models are trained in a self-supervised fashion: given a stream of observations (e.g., consecutive video frames), part of the input is masked or withheld, and the model learns to predict the missing or upcoming content—typically in a learned representation space rather than at the level of raw pixels. Repeated over massive amounts of data, this yields representations that capture how observations tend to evolve over time.

But is this the only path forward?

To answer this, it is important to revisit what we mean by a world model. In much of the AI literature, the term is used in a relatively narrow sense: a system that predicts future observations.

In contrast, cognitive science adopts a broader perspective. Human mental models are not limited to prediction—they also support explanation, abstraction, and counterfactual reasoning. For example, the theory theory suggests that humans understand the world through intuitive theories. These range from simple rules (e.g., “gravity makes objects fall”) to more structured representations, such as mathematical models. More abstract forms of world models have also been explored in AI, including Bayesian networks and program-like representations [14,15].

Another important observation in humans is that mental models are not learned passively (which strongly contrasts from how most recent approaches train world models).

While children do learn from observation, a key component of acquiring a robust world model is interaction. Humans actively explore their environment: they experiment, observe outcomes, and refine their internal models accordingly.

This process is often described as curiosity-driven exploration—a form of behavior aimed at gathering informative experiences rather than maximizing immediate reward [16,17,18]. This loop of exploration and model refinement is not unique to artificial agents. It mirrors what developmental psychologists argue is a core driver of children’s cognitive growth [19]—and, at a higher level, it is precisely what scientists do: form hypotheses, design experiments to test them, and revise their understanding accordingly.

A similar insight has emerged in AI. In reinforcement learning, purely random exploration is often insufficient to learn accurate models of the environment. Instead, agents benefit from exploration strategies that prioritize informative or novel situations, enabling more efficient learning of world dynamics.

In that sense, building world models is not just a matter of prediction—it fundamentally relies on how data is collected. Effective world models emerge from meaningful, information-driven interaction with the environment.

3) WorldLLM: curious LLMs that explore and refine their world model

In our team, studying curiosity-driven exploration has been a central topic, from modelling curiosity in humans (and in particular children) to designing curious artificial agents. In recent years, we’ve proposed several approaches for augmenting LLMs with curiosity-based exploration [20,21], allowing them to set and learn their own goals (i.e., turning them into autotelic agents [21,22]). Building on this line of work, WorldLLM—led by Guillaume Levy and colleagues—asks whether an LLM can go a step further and actively build an abstract model of its environment through interaction, by combining curiosity-driven exploration with language-based reasoning.

More specifically, WorldLLM embodies an LLM in an agent that acts in an environment, collects experiences it struggles to predict, and generates natural-language theories to explain—and ultimately better predict—what it observes. Crucially, the resulting world model is not a neural network fitted to data, but a set of abstract theories expressed in natural language: human-readable, interpretable, and stated explicitly rather than buried in a model’s weights.

In this respect, the agent behaves much like a scientist: it confronts the limits of its current understanding, formulates explanatory hypotheses, and tests them against reality. This framing connects to a recent wave of work on artificial automated scientists, where LLMs autonomously generate ideas, run experiments, and write up findings [23,24]. WorldLLM shares the same spirit, but targets a more fundamental ingredient: building and refining an abstract model of how the world works.

This makes it worth being explicit about what WorldLLM is and is not. It is not a new LLM, nor a new kind of LLM—it does not modify the architecture or training of the underlying language model. It is an agentic architecture wrapped around an existing LLM. And unlike most LLM agents today, which are built to write code or carry out user-defined tasks, WorldLLM’s agent has a different purpose entirely: to formulate hypotheses about its environment and put them to the test through active experimentation. Its capability lies not in any single model call, but in the loop of acting, observing, and revising.

To do this, WorldLLM frames the acquisition of a world model as a process of Bayesian inference over theories. Rather than learning a single predictive model, the system maintains a distribution over possible explanations of the environment, where each theory is a structured, language-based hypothesis about how the world behaves. The space of such theories is vast—and this is where the LLM becomes essential.

The LLM plays a dual role. On one hand, it predicts observations, conditioning on a candidate theory supplied in its prompt. On the other, it acts as a proposal mechanism, generating new theories meant to explain past observations—effectively serving as a learned prior over plausible, structured hypotheses.

These theories are not judged in the abstract: they are tested through interaction. The agent collects new observations, and the LLM assesses how well each theory accounts for them. Reinforcement learning drives this data collection, and—critically—the exploration is curiosity-driven. Rather than passively observing, the agent is rewarded for seeking out the most informative situations: those where competing theories disagree, or where the current model is most uncertain. This closes a learning loop that is, again, fundamentally agentic: propose theories → test them through interaction → observe outcomes → update beliefs

Over time, the agent sharpens its distribution over theories, favoring those that help the LLM better explain what it encounters. In doing so, WorldLLM ties language-based reasoning to environment interaction, grounding theories expressed in language by testing them through experience.

When evaluated in a textual environment where an agent must manipulate and combine objects, WorldLLM not only improves predictive accuracy but also yields human-interpretable theories of the environment’s dynamics, making explicit what the system has come to understand.

4) From prediction to understanding

World models have long been framed as predictive tools: systems that learn to anticipate what comes next. But as we have seen, prediction alone may not be enough.

Humans do not just passively model the world—we form hypotheses, test them through interaction, and refine them over time. Our understanding emerges not only from data, but from how we choose to collect it.

Rather than treating LLMs as static predictors, WorldLLM embeds them in a loop of hypothesis generation, experimentation, and belief updating. The LLM is no longer just a model of text—it becomes part of a system that actively seeks to understand the world. As mentioned at the beginning of Section 3, this is part of a broader line of work in our team on autotelic LLMs: studying how language models can explore, set their own goals, and learn to control their environment rather than merely describe it [7,20]. WorldLLM’s focus on building world models complements these earlier efforts to ground LLMs through interaction and to guide their exploration of large goal spaces—together sketching a view of LLMs as agents that learn from, and act upon, the world.

However, this perspective raises several open questions. Can language-based theories capture the richness of physical and causal dynamics?
Should future AI systems rely more on interaction and exploration, rather than ever-larger static datasets? Can current approaches used to build world models be used to capture models of others (i.e., Theory Of Mind)?

Acknowledgements

Thank you to Pierre-Yves Oudeyer for his thoughtful feedback and references.

References

[1] Craik, K. J. W. 1943. The Nature of Explanation. Cambridge University Press.

[2] Johnson-Laird, Philip Nicholas. 1983. Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness. Harvard University Press.

[3] Assran, Mido, Adrien Bardes, David Fan, et al. 2025. “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.”

[4] Bardes, Adrien, Jean Ponce, et Yann LeCun. 2023. “MC-JEPA: A Joint-Embedding Predictive Architecture for Self-Supervised Learning of Motion and Content Features.”

[5] Nam, Heejeong, Quentin Le Lidec, Lucas Maes, Yann LeCun, et Randall Balestriero. 2026. “Causal-JEPA: Learning World Models through Object-Level Latent Interventions”

[6] Zhou, Gaoyue, Hengkai Pan, Yann LeCun, et Lerrel Pinto. 2025. “DINO-WM: World Models on Pre-Trained Visual Features Enable Zero-Shot Planning.” Forty-second International Conference on Machine Learning.

[7] Romac, Clément, Thomas Carta, Pierre-Yves Oudeyer. “Can AIs understand our world? Functionally grounding LLMs in interactive environments.” https://developmentalsystems.org/glam.

[8] Zahavy, Tom. 2026. “LLMs can’t jump.”

[9] LeCun, Y. 2022. “A path towards autonomous machine intelligence version 0.9.2.” Open Review, 62(1), 1–62.

[10] Werbos, Paul J. 1987. “Learning How the World Works: Specifications for Predictive Networks in Robots and Brains.” Proceedings of the IEEE Conference on Systems, Man and Cybernetics.

[11] Sutton, Richard S. 1990. “Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming.” Proceedings of the Seventh International Conference on Machine Learning (ICML).

[12] Moore, Andrew W., et Christopher G. Atkeson. 1993. “Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time.” Machine Learning 13(1): 103–130.

[13] Moerland, Thomas M., Joost Broekens, Aske Plaat, et Catholijn M. Jonker. 2023. “Model-Based Reinforcement Learning: A Survey.” Foundations and Trends in Machine Learning 16(1): 1–118.

[14] Piriyakulkij, Wasu T., Cassidy Langenfeld, Tuan A. Le, et Kevin Ellis. 2024. “Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning.” Advances in Neural Information Processing Systems 37: 53102–37.

[15] Tsividis, Pedro A., Joao Loula, Jake Burga, et al. 2021. “Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning.”

[16] Baldassarre, Gianluca, et Marco Mirolli, éd. 2013. Intrinsically Motivated Learning in Natural and Artificial Systems. Springer.

[17] Oudeyer, Pierre-Yves, et Frederic Kaplan. 2007. “What Is Intrinsic Motivation? A Typology of Computational Approaches.” Frontiers in Neurorobotics 1.

[18] Schmidhuber, Jürgen. 1991. “A possibility for implementing curiosity and boredom in model-building neural controllers.” Proceedings of the first international conference on simulation of adaptive behavior on From animals to animats.

[19] Gopnik, Alison, Andrew N. Meltzoff, et Patricia K. Kuhl. 1999. The Scientist in the Crib: Minds, Brains, and How Children Learn. William Morrow & Company.

[20] Gaven, Loris, Thomas Carta, Clément Romac, Cédric Colas, Sylvain Lamprier, Olivier Sigaud, et Pierre-Yves Oudeyer. 2025. “MAGELLAN: Metacognitive Predictions of Learning Progress Guide Autotelic LLM Agents in Large Goal Spaces.” Proceedings of the 42nd International Conference on Machine Learning (ICML’25), Vol. 267.

[21] Colas, Cédric, Tristan Karch, Olivier Sigaud, et Pierre-Yves Oudeyer. 2022. “Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: A Short Survey.” Journal of Artificial Intelligence Research 74.

[22] Steels, Luc. 2004. “The Autotelic Principle.” In Embodied Artificial Intelligence: International Seminar, Dagstuhl Castle, Germany, July 7-11, 2003. Revised Papers, edited by Fumiya Iida, Rolf Pfeifer, Luc Steels, and Yasuo Kuniyoshi. Springer.

[23] Lu, Chris, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, et David Ha. 2024. “The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery.”. arXiv:2408.06292

[24] Yamada, Yutaro, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, et David Ha. 2025. “The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search.”. arXiv:2504.08066

Learning together: how others can fuel (or hinder) our curiosity

Fri, 24 Apr 2026 08:00:00 +0000

A story of “motivating” cakes

A few days ago, our team gathered around a table for a collective writing session — a “blogathon,” as we called it. Each of us had to write a post on a research topic related to the team’s focus. The goal was to learn or practice writing scientific content in this format. Between bites of the small cakes brought by a colleague, I started typing my first lines… a situation that perfectly illustrated, in real conditions, the very topic I wanted to address.

Why am I writing this article? To learn how to write a blog post? To share my research? To deepen my own understanding of the literature? To be part of a team dynamic initiated by our director? Because my colleagues were there too, writing, creating a focused and supportive working atmosphere around me?

The honest answer: probably all of the above. And it is precisely this overlap of motivations—reflecting many everyday activities—that leads us to a fundamental question in cognitive and educational sciences: to what extent do others—their mere presence, their gaze, their activity—influence our motivation to learn?

Intrinsic and extrinsic motivation: an inseparable duo

To address this question, we first need to establish some conceptual foundations. Since the seminal work of E. Deci (1975), two major types of motivation have been distinguished.
Extrinsic motivation refers to actions driven by external rewards or the avoidance of punishment—such as grades, salary, or social approval.
Intrinsic motivation, on the other hand, refers to engaging in an activity for the inherent pleasure and satisfaction it provides, without any obvious external incentive.

Rather than being strictly opposed, these two forms of motivation often coexist and interact in most of our behaviors. Among the different types of intrinsic motivation, we find the desire to learn—also known as curiosity —which can be defined as a dynamic process that drives individuals toward spontaneous exploration and the active search for new knowledge. Curiosity is therefore a key driver of (self-directed) learning, present from birth, guiding attention, sustaining engagement, and enhancing memory retention.

Understanding this learning-related curiosity lies at the core of the work conducted by the Flowers team at INRIA Bordeaux Sud-Ouest (Team website). One of their main goals is to understand how curiosity emerges, develops, and can be supported in real-world learning environments. Understanding these mechanisms is crucial for designing more effective educational tools, particularly for children.

But do situations truly exist where we can learn “alone,” in a complete social vacuum?

To understand the links between motivation and environment, Self-Determination Theory (SDT) developed by Deci and Ryan (1985) provides a key theoretical framework (Learn more). This macro-theory aims to identify what enhances—or disrupts—intrinsic motivation. It highlights three fundamental psychological needs whose satisfaction supports intrinsic motivation:
• Autonomy: feeling that one is the origin of one’s own actions and choices,
• Competence: experiencing a sense of effectiveness,
• Relatedness: feeling connected to others, recognized and accepted.

One of its sub-theories, Cognitive Evaluation Theory (CET), focuses specifically on the role of social context. According to CET, a social environment that supports these three needs enhances intrinsic motivation.

Although curiosity is a specific form of intrinsic motivation, it has so far been relatively understudied within the SDT framework. Nevertheless, CET could still apply, as curiosity may also be sensitive to social context—particularly in educational settings where social interactions can act as a form of extrinsic influence (Deci, Koestner & Ryan, 2001). Positive feedback, trustful relationships with peers or teachers, and a sense of being understood and valued are all factors that can foster the desire to learn. Conversely, environments perceived as coercive, highly competitive, or strongly focused on evaluation may undermine this motivation.

Thus, curiosity may not be purely individual—it may instead be deeply shaped by the social context in which it unfolds. The question remains: through which mechanisms, and under what conditions, does the presence and influence of others enhance—or hinder—our desire to learn? To explore this, we must go back to the roots of social influence on human learning—long before school, and even before social pressure or rewards become meaningful. From the very first months of life, learning occurs not only through exploration, but also through observing others.

The influence of others on our learning does not begin at school. It starts at birth—or even earlier. Early learning in infants relies heavily on imitation: observing to understand, reproducing to internalize. These processes continue and become more specialized throughout development and adulthood in domains such as sports, music, or craftsmanship.

As described in Bandura’s Social Learning Theory (1977), we learn not only through direct experience, but also by observing others. This form of learning is so powerful that it can lead to overimitation: we reproduce actions even when they are not functionally necessary—simply because others performed them (Hoehl et al., 2019).

This phenomenon likely serves two functions: a cognitive and instrumental one, supporting learning and understanding others’ actions, and a social one, enabling affiliation and communication. This highlights how deeply our brains are tuned to social information.

Moreover, to briefly address brain networks, this imitation system is notably supported by the mirror neuron system, initially identified in the 1990s in non-human primates (Rizzolatti & Craighero, 2004), and whose functional scope (in humans) extends beyond motor areas to include regions involved in understanding others’ intentions and emotions. Indeed, social learning is not limited to the motor domain. It extends to our beliefs, values, judgments, and ultimately our decisions. We continuously revise our knowledge in light of what others do or think—a mechanism that Bayesian theories of reinforcement learning model in terms of probabilistic inference, where social signals constitute essential information to integrate in order to reduce uncertainty (see, for example, Hofmans and van den Bos, 2022). Among the explicit forms of social influence that enable this updating of our knowledge is conformity.

Top figure: from Arioli and Canessa (2019) in Human Brain Mapping: brain network involved in social learning (learning by observation), including premotor cortex, parietal areas, inferior frontal gyrus, superior temporal sulcus, and occipital regions.
Bottom figure: from Hofmans and van den Bos (2022) in Developmental Cognitive Neuroscience: Bayesian updating of beliefs as a function of the social characteristics of the source. Social information is weighted according to its perceived reliability (expertise, status, proximity): it has a weaker impact when it comes from a low-credibility source (top diagram) and a stronger impact when it comes from a reliable source (bottom diagram). This differential weighting leads to a greater or lesser revision of the initial belief.

Learning under group pressure: conformity

Conformity is our tendency to align our judgments and behaviors with those of the group. Recently synthesized in a systematic review written by Emilie Caspar and myself (Tricoche and Caspar, currently under publication, preprint available), this body of work (going back to the first evidence provided by S. Asch in the 1950s) shows that we can go so far as to contradict our own sensory perception, beliefs, and memory in order to conform to the majority.

Two main levels of influence can be distinguished to explain conformity:
• Normative influence: we conform in order to be accepted, avoid rejection, and belong to the group,
• Informational influence: we conform because we consider others to be a reliable source of information, especially in situations of uncertainty.

These two levels of influence have direct implications for learning. If a child or an adult learns in a social environment (which is the case in the majority of school and professional situations), their responses will inevitably be influenced by what they perceive of others’ beliefs and knowledge. This can promote a rapid diffusion of knowledge (in the case of benevolent informational influence), or, on the contrary, suppress autonomous exploration if normative pressure is too strong, or even coercive—thereby undermining precisely the autonomy that SDT identifies as essential to intrinsic motivation.

The forms of social influence and social context described so far assume an interaction, or at the very least information coming from others. But what happens when no explicit information is transmitted, when the other person is simply there?

Learning through the mere presence of others

One thing is certain: others are always there. We learn surrounded by our peers, our teachers, our colleagues—rarely alone in front of a task and sheltered from any gaze. This is why it is important to ask whether this presence, even silent, even passive, would be enough to modify our behaviors and our (curiosity-driven) learning.

The first proof-of-concept evidence regarding the mere presence phenomenon goes back to the earliest observations and experiments by Triplett in 1898, who used a device—with a somewhat archaic appearance—to show that children performed better in a motor task when they were in pairs than when they were alone. Subsequently, many studies validated this positive effect of the presence of others, but also showed that it could impair performance. Researchers such as Zajonc (1965) and Baron (1968) proposed theoretical frameworks to explain this duality, which can be summarized—somewhat simplistically—as follows: the mere presence of others acts as an attentional distractor, improving performance on well-mastered tasks, but potentially disrupting success on new or complex tasks.

Three forms of social presence are typically distinguished:
• Passive audience: someone is there, present, but does not necessarily observe,
• Evaluation: someone is observing with a potentially evaluative gaze,
• Coaction: several individuals perform the same task at the same time, without direct interaction—thus distinguishing it from cooperation or competition, which involve interdependence between individuals and introduce a different level of social pressure.

For more exhaustive content on the mere presence effect

Top figure: experimental device used by Triplett in his 1898 study on the mere presence effect in children. It was a device resembling a fishing rod, in which children had to turn a reel as quickly as possible.
Bottom figure: the three types of mere presence represented in an example of a racing situation. One can distinguish passive audience, evaluation, and coaction.

This basic form of social influence seems to act on every type of behavior, from the most elementary (e.g., eating, moving the eyes) to the most complex (e.g., memorizing, deciding, counting) (Guérin, 2010). What about our learning and the associated intrinsic motivation, particularly in educational contexts?

I carried out a survey of the PubMed database, crossing terms related to mere presence on the one hand, and intrinsic motivation and learning on the other (studies in typically developing humans, considering original studies from the last 20 years, as well as reviews and meta-analyses from the last 10 years), and arrived at a rather surprising conclusion: the field is almost empty.

A few studies related to educational skills, in children and adolescents, show effects of social presence, notably on reasoning (Wolf et al., 2015), creativity (Camarda et al., 2021), or numerical and language skills (Tricoche et al., 2021). In addition, neuroimaging studies seem to indicate that the presence of another person does not act only on attentional processes, but also on motivational and emotional processes, even in the absence of any explicit reward (Tricoche et al., 2023). In other words, others, through their mere presence, could be a form of implicit extrinsic motivation, capable of influencing intrinsic motivation, without individuals necessarily being aware of it. Likewise, brain regions involved in social cognition are activated when others are present, in order to enable the mentalizing process. Indeed, as soon as another human being enters our environment, our mentalizing processes are activated—this capacity to infer the mental states of others, commonly referred to as theory of mind. “They can see me,” “what do they think of my performance?”, “do they like me?”: these questions, even when formulated unconsciously, mobilize cognitive and emotional resources that modify our engagement in the task. Others then become associated with a form of implicit social reward—being well perceived, approved of, accepted.

Top figure: from Arioli and Canessa (2019) in Human Brain Mapping: brain network involved in mentalizing (theory of mind), including mainly the temporo-parietal junction, the medial prefrontal cortex, and the precuneus extending into the posterior cingulate cortex.
Bottom figure: from Tricoche et al. (2023) in NeuroImage: numerous regions are more activated in social presence than in the alone condition during the completion of cognitive tasks. These regions seem to belong to the attentional network, the mentalizing network, and the motivational network (the ventral striatum, VS, in particular).

The Flowers team in this landscape: the first milestones

It is precisely on this still little explored link between learning, motivation, and social presence that the Flowers team is beginning to raise empirical questions.

The team is known for its work on the computational mechanisms of curiosity and intrinsic motivation, notably in educational settings, some of which have been tested in schools. Paradigms have been developed specifically to address the understanding of factors related to curiosity, with a specific focus on the learning progress hypothesis (i.e., an intrinsic signal that guides exploration toward the activities that are the most informative and those that make us progress the most). Among them are so-called “free exploration” (gamified) tasks in which participants discover and freely explore an environment rich in learning activities with varying levels of difficulty (Ten et al., 2021; Poli et al., 2025; Serko, Leonard & Ruggeri, 2025). In these paradigms, participants can choose to continue the same activity or switch to another level of difficulty (simpler or more complex).

However, up to now, this work has mainly focused on the individual dynamics of curiosity. The explicit integration of social factors therefore constitutes a natural extension of this research program.

Paradigms studying how individuals orient their exploration choices as a function of learning progress.
Top figure: from Ten et al. (2021) in Nature Communications: “Monster Task,” in which there are different families of monsters with hidden rules/behaviors that the participant freely discovers. At each trial, the participant chooses a monster, formulates a hypothesis about its food preference, then receives feedback.
Bottom figure: from Poli et al. (2025) in Child Development: in this exploration task, the participant interacts with characters by guessing the place where they are hiding.

A first step toward understanding social influence on curiosity-driven learning was taken through a study conducted by C. Devaux et al. (currently under publication). The results suggest that during metacognitive intervention sessions aimed at improving curiosity in learning, the type of adult leading the session had a different impact. Indeed, the presence of the regular teacher, compared with that of the experimenter, during these sessions, reduced pupils’ learning performance and curiosity. In addition, subjective questionnaires revealed an increased cognitive load. Possibly, pre-existing social dynamics—implicit expectations, hierarchical relationship, interactional habits—may influence the way pupils mobilize their cognitive and motivational resources. One interpretative hypothesis is that the social novelty embodied by the experimenter could act as an implicit motivational signal.

These first field findings open the way to a series of questions that I wish to address in the context of my postdoctoral work within the Flowers team, notably in collaboration with J. Perez (PhD student within the team; Perez et al., currently under publication, preprint available): is the mere presence of another individual enough for curiosity to intensify or deteriorate? Does the presence of a peer (classmate) have the same effect as that of an adult (teacher)? And how do these influences differentially affect learning according to age, educational context (e.g., traditional school versus Montessori pedagogy), or the child’s profile, whether in terms of cultural background, personality traits, or relationship with their social surroundings?

Conclusion

Thus, learning is not a solitary process. Others are not mere distractors or sources of pressure: they are also catalysts, mirrors, and silent partners of our curiosity. Understanding how and under which conditions they enhance—rather than suppress—our intrinsic motivation is a major scientific challenge, and a concrete educational challenge for all those who design learning environments and intervene in them.

References

• Arioli, M., & Canessa, N. (2019). Neural processing of social interaction: Coordinate-based meta-analytic evidence from human neuroimaging studies. Human Brain Mapping, 40(13), 3712–3737. https://doi.org/10.1002/hbm.24627
• Asch, S. E. (1951). Effects of group pressure upon the modification and distortion of judgments. Groups, leadership and men; research in human relations, pp. 177–190. Carnegie Press. https://doi.org/10.1525/9780520313514-017
• Bandura, A. (1977). Social Learning Theory. Prentice Hall.
• Camarda, A., Bouhours, L., Osmont, A., Le Masson, P., Weil, B., Borst, G., & Cassotti, M. (2021). Opposite Effect of Social Evaluation on Creative Idea Generation in Early and Middle Adolescents. Creativity Research Journal, 33(4), 399–410. https://doi.org/10.1080/10400419.2021.1902174
• Deci, E. L. (1975). Intrinsic motivation. New York, Plenum Press. https://doi.org/10.1007/978-1-4613-4446-9
• Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Springer NY. https://doi.org/10.1007/978-1-4899-2271-7
• Deci, E. L., Koestner, R., & Ryan, R. M. (2001). Extrinsic Rewards and Intrinsic Motivation in Education: Reconsidered Once Again. Review of Educational Research, 71(1), 1-27. https://doi.org/10.3102/00346543071001001
• Devaux, C., Abdelghani, R., Sauzéon, H., & Oudeyer, P-Y. (Under review). Metacognitive training of children’s curiosity in schools: investigating a teacher-led intervention.
• Guerin, B. (2010). Social facilitation. Cambridge University Press. https://doi.org/10.1017/CBO9780511628214
• Perez, J., Derex, M., Oudeyer, P-Y., & Moulin-Frier, C. Intrinsic motivation is key to understanding peer cultures. preprint
• Poli, F., Meyer, M., Mars, R. B., & Hunnius, S. (2025). Exploration in 4-year-old children is guided by learning progress and novelty. Child Development, 96:192-202. https://doi.org/10.1111/cdev.14158
• Rizzolatti, G., & Craighero, L. (2004). The mirror-neuron system. Annual Review Neuroscience, 27:169-192. https://doi.org/10.1146/annurev.neuro.27.070203.144230
• Serko, D., Leonard, J., & Ruggeri, A. (2025). Children strategically decide what to practice. Child Development, 96:1619-1631. https://doi.org/10.1111/cdev.14268
• Ten, A., Kaushik, P., Oudeyer, PY., Gottlieb, J. (2021). Humans monitor learning progress in curiosity-driven exploration. Nature Communications, 12, 5972. https://doi.org/10.1038/s41467-021-26196-w
• Tricoche, L., Monfardini, E., Reynaud, A. J., Epinat-Duclos, J., Pélisson, D., Prado, J., & Meunier, M. (2021). Peer presence effect on numerosity and phonological comparisons in 4th graders: When working with a schoolmate makes children more adult-like. Biology, 10(902), 643–662. https://doi.org/10.3390/biology10090902
• Tricoche, L., Pélisson, D., Longo, L., Kound, E., Poisson, A., Prado, J. & Meunier, M. (2023). Task-independent neural bases of peer presence effect on cognition in children and adults. NeuroImage, 277. https://doi.org/10.1016j.neuroimage.2023.120247
• Tricoche, L., & Caspar, E. (Under review). The power of social influence on (non-)moral decision-making and its resistance: A scoping review of behavioral effects and neural mechanisms. preprint
• Triplett, N. (1898). The dynamogenic factors in pacemaking and competition. The American Journal of Psychology, 9(4), 507–533. https://doi.org/10.2307/1412188
• Wolf, L. K., Bazargani, N., Kilford, E. J., Dumontheil, I., Blakemore, S-J. (2015). The audience effect in adolescence depends on who’s looking over your shoulder. Journal of Adolescence, 43: 5-14. http://dx.doi.org/10.1016/j.adolescence.2015.05.003
• Zajonc, R. (1965). Social facilitation. Science, 149, 269–274. https://doi.org/10.1126/science.149.3681.269

The Pleasure of Getting Better: Is Learning a Basic Need?

Thu, 09 Apr 2026 00:00:00 +0000

When LLMs Play the Telephone Game: An Interactive Tool

Mon, 09 Mar 2026 00:00:00 +0000

The Phylogenetics of Artifacts - A deep dive into the evolution of cultural objects, artificial life forms and language models

Fri, 27 Feb 2026 00:00:00 +0000

The aim of this blog is to show that phylogenetic methods — traditionally used to reconstruct the evolutionary history of biological species — can be applied far beyond their original scope. This idea was first theorised by Dawkins in 1976 [1] and has since been demonstrated on cultural artifacts by researchers such as Julien d’Huy [2]. Here, we take it a step further and apply it to the intelligence of Large Language Models (LLMs).

Estimated dendrogram of some LLMs from OpenAI, Anthropic, Google and Mistral.

This is a particularly timely question in a world where the number of LLMs is growing at an extraordinary pace — around 300 new models are published every day on the HuggingFace hub — and where transparency about training details is increasingly limited. Reconstructing the phylogenetic history of models in a black-box setting, without any prior information beyond interacting with them, would provide a powerful tool to map the landscape of modern AI, shed light on lineages that are not always made explicit, and inform the evaluation of AI capabilities — with potential applications to AI safety and the monitoring of emerging risks.

Part I: What is Phylogenetics ?

Phylogenetics is the study of evolutionary relationships among organisms. It involves reconstructing the evolutionary history and relatedness of different species or populations, by analyzing their characteristics and determining how they descended from common ancestors.

The main goal is to understand how different organisms are connected through evolution, typically represented in tree-like diagrams called phylogenetic trees. These trees illustrate which species share recent common ancestors, how lineages branched over time, and the sequence in which different groups evolved.

Populations of individuals that frequently reproduce with each other (sexual reproduction) or share genetic material (as bacteria often do) tend to maintain some homogeneity in their genetic data. However, when a population splits in two — think of insects in a forest divided by a fire, suddenly unable to reach one another — the genetic sharing stops, and the two groups begin evolving much more independently. When they eventually become too genetically different to reproduce with each other, they are said to have speciated: they are now two distinct species. The scenario described above — separation by geography — is known as allopatric speciation, and it’s just one of several ways speciation can occur.

With that foundation in place, let’s look at how we actually reconstruct the evolutionary history of living species from the speciations that shaped them and then how these ideas can be relevant to study the evolution of cultural artefacts and LLMs.

Methods to reconstruct phylogenetic trees in biological systems

This blog will only explore one method of reconstructing biological evolution through genetic material to give a concrete example of how such methods work.

Given a set of populations (species, animal races, communities, etc.) that we want to reconstruct the genetic evolutionary history of, we first need to select some genes common to all of them as a basis for comparison. To make this concrete, let’s imagine we want to reconstruct the evolutionary history of five cat breeds — the Persian, the Siamese, the Maine Coon, the Bengal, and the British Shorthair. One could say these are not distinct species, but the same algorithm applies to both breeds and species and it makes a great illustration example.

Cat breeds used for the study - image generated by Gemini for the sake of illustration in this blog.

Let’s walk through each step of the phylogenetic pipeline to understand which breeds share a recent common history and which diverged long ago.

How to select the genes ?

We need genes that varied moderately at the timescale of the study. For example, if we study a set of cat breeds, genes related to reproduction may not have varied much between breeds, offering too little variation to distinguish between them. On the other hand, genes related to fur color may offer too much variance, making all breeds appear very different from each other and the results too noisy to interpret. It is therefore important to choose genes that show moderate variance and that capture the evolutionary signal we are looking for.

To go back to our population of cats, relevant genes could be related to eye color, hair length, or overall skull morphology — traits that vary enough between breeds to be informative, but not so much as to obscure the underlying evolutionary relationships. Let’s say we select three genes for our toy study: Gene C (eye color), Gene E (ear length), and Gene S (skull morphology). Each of these genes has a small number of possible variants (alleles). For instance, Gene C might have three alleles in this toy example: blue, copper, and green; Gene E might have three: short, medium, and long; and Gene S might have two: round and elongated. To be fair, in practice these traits are not the result of a single gene or allele but usually a combination of several. For the sake of simplicity, this blog assumes a single gene per trait and uses this simplification only to illustrate the process of building a phylogenetic tree with a small and interpretable number of variables.

All the genes discussed here are functional genes, meaning they directly impact the physical attributes of the cat. In practice, many modern phylogenetic studies use DNA segments that are non-functional — not expressed, and not tied to any physical attribute. For the sake of clarity, this blog illustrates the pipeline using genes related to physical traits because it makes the process visual and easy to follow. That said, it is important to note that phylogenetic studies are grounded in the genotype of an individual (its genes) rather than its phenotype (its physical appearance), since individuals can converge on very similar appearances while having very different evolutionary histories.

Gathering genetic information

Once the genes are selected, we can gather individuals from each population of cats and measure which version of each gene they carry (called an allele). For example, in a population of Persians, we might sample 100 individuals and find that 95% carry the round skull allele and only 5% carry the elongated allele for Gene S, while in the Siamese population the split might be 15% round and 85% elongated. This is stored in a population function P(allele | gene) — the probability of finding a given allele for a given gene when sampling a random individual from that population.

Continuing with our example, here is what the population functions might look like for all five breeds across our three selected genes:

Breed	Gene C (blue / copper / green)	Gene E (short / medium / long)	Gene S (round / elongated)
Persian	0.05 / 0.85 / 0.10	0.90 / 0.10 / 0.00	0.95 / 0.05
Siamese	0.98 / 0.02 / 0.00	0.00 / 0.10 / 0.90	0.15 / 0.85
Maine Coon	0.05 / 0.60 / 0.35	0.00 / 0.15 / 0.85	0.30 / 0.70
Bengal	0.05 / 0.55 / 0.40	0.10 / 0.75 / 0.15	0.25 / 0.75
British Shorthair	0.03 / 0.90 / 0.07	0.15 / 0.80 / 0.05	0.85 / 0.15

A population function is estimated for each breed, and these distributions are then compared to measure how similar or distant the breeds are from one another.

Population comparison

Once a population function is estimated for each population studied, they are compared using a mathematical formula, which can vary between studies. For this blog we are going to use one of the most classic: Nei’s genetic distance [3], defining a similarity matrix S and a distance matrix D:

$S(P_1, P_2) = \frac{\sum_{g \in G} \sum_{a \in A_g} P_1(a|g) \cdot P_2(a|g)}{\sqrt{\left(\sum_{g \in G} \sum_{a \in A_g} P_1(a|g)^2\right) \left(\sum_{g \in G} \sum_{a \in A_g} P_2(a|g)^2\right)}}$

$D(P_1,P_2) = -\log(S(P_1,P_2))$

Where:

$P_1$ and $P_2$ are two populations (or LLMs), seen as probability distributions of alleles given a gene
$G$ is the set of genes considered
$A_g$ is the set of possible alleles for gene $g$
$S$ is the similarity matrix, bounded in $[0, 1]$
$D$ is the distance matrix, in $[0,+\infty]$

To put it simply, the similarity formula compares the probability of seeing the same allele appear in both populations for the same gene, and normalises it by the probability of seeing that allele in each population individually. This produces a measure of how similar the two population functions are. The distance matrix D is then derived from the similarity matrix and used to plot the phylogenetic tree.

Let’s see what this similarity matrix looks like between our cat breeds:

	Persian	Siamese	Maine Coon	Bengal	British Shorthair
Persian	1.00	0.10	0.78	0.41	0.69
Siamese	0.10	1.00	0.35	0.47	0.22
Maine Coon	0.78	0.35	1.00	0.73	0.57
Bengal	0.41	0.47	0.73	1.00	0.77
British Shorthair	0.69	0.22	0.57	0.77	1.00

This matrix represents the similarity between cat breeds as seen through the lens of the selected genes. Persians and Siamese, for instance, are the most genetically distant pair in our study (0.10 similarity), while British Shorthairs and Persians appear considerably closer (0.69 similarity).

Phylogenetic tree

The concept behind plotting a phylogenetic tree is that populations which speciated recently will remain fairly similar, while populations that speciated long ago will have diverged significantly as mutations accumulate over time. This is why gene selection matters so much: genes that mutate too fast or too slowly will not efficiently capture the evolutionary history of the species studied. Several algorithms have been proposed to compute a tree from a distance matrix; the one discussed here is one of the most classic: the Neighbour-Joining technique (NJ tree) [4]. The goal is not to give a full tutorial on the NJ algorithm but to provide an overview of the approach.

The core idea is that populations which diverged most recently should be the most genetically similar. The algorithm identifies the two populations that are closest to each other relative to their distance to all other populations — not simply the two with the smallest absolute distance. These two populations are then grouped into a clade, representing their hypothetical common ancestor. Once this clade is formed, the two original populations are removed from the distance matrix and replaced by their common ancestor, whose distance to every remaining population is estimated as a weighted average of the distances from the two child populations. The matrix now has one fewer entry (n−1 instead of n). This process repeats — finding the next closest pair, grouping them, and reducing the matrix — until only two populations remain, joined at the root of the tree: the ancestor of all.

This method makes it possible to build a phylogenetic tree representing one possible version of the evolutionary history of these species, seen through the prism of the selected genes.

Back to our cats, here is the NJ tree for the five breeds:

        ┌──────────────────── Siamese 🐱
───┤Clade 4
        │   ┌──────────────── Maine Coon 🦁
        └─┤Clade 3
             │   ┌──────────── Bengal 🐆
             └─┤Clade 2
                 │  ┌──────── British Shorthair 🐈
                 └─┤Clade 1
                     └──────── Persian 🐈‍⬛

This is compatible with the genetic literature: British Shorthair and Persian speciated recently relative to Maine Coon or Siamese. The Bengal, however, is a special case — it is a crossbreed between Asian leopard cats and domestic European cats, meaning its evolutionary history cannot be cleanly captured by a tree structure.

This highlights a broader limitation of the approach: it rests on the assumption that evolution follows a tree-like structure, with limited interaction between lineages such as horizontal gene transfer or complex interbreeding. As domesticated animals, cats have been heavily shaped by human intervention, making their evolutionary history particularly tangled. Phylogenetic methods like the one presented here are most reliable when applied to species whose evolution more closely follows these theoretical principles.

Part 2: Theory of Evolution

We have seen how to build phylogenetic trees using a simple method. But why does it actually work? What properties of DNA make it such a reliable marker of evolutionary history, and could these properties exist in non-biological objects? This section explores the theoretical foundations of phylogenetics, which we will later use as a framework to study the evolution of cultural artifacts and language models.

DNA: the functional and archeological support for evolution

Studying past evolution revolves around finding markers of that evolution that are still accessible today. In biology, these markers are DNA mutations. When reproducing, mutations are introduced into the DNA of the offspring and may then be passed on to future generations. These mutations come in two broad flavors:

Mutations under selective pressure affect the fitness of the individual in its environment — making it more or less likely to survive and reproduce. A skin pattern that improves camouflage, or the ability to resist extreme temperatures, are classic examples. Beneficial mutations of this kind tend to spread rapidly through a population, as individuals carrying them reproduce more successfully. Harmful ones, on the other hand, are quickly eliminated: individuals who develop them tend to die before reproducing or fail to find mates, and so the mutation disappears from the gene pool. Because of this, genes that encode vital features — heart architecture, reproductive organs — show very little variation across individuals: any mutation in these genes is almost certainly harmful and gets weeded out fast.

Neutral mutations do not affect the fitness of the individual. A DNA sequence that is not expressed, or one that produces an equivalent protein, or traits like eye color that have little impact on survival or reproduction — these can accumulate freely across generations. Because they are not filtered by selection, they evolve at a much steadier and more predictable rate, making them the most useful markers for tracing evolutionary history. The further back two populations diverged, the more neutral mutations will have accumulated between them. Because neutral mutations are not filtered by selection, their accumulation is driven primarily by random chance — a process known as genetic drift. This randomness is actually what makes neutral genes such reliable evolutionary clocks: drift operates at a relatively steady rate, unlike selection which is episodic and environment-dependent.

This distinction is crucial: genes under strong selective pressure evolve slowly, while neutral genes evolve more steadily. Phylogenetics primarily relies on the latter to reconstruct the branching history of species especially for smaller time scale studies (as discussed above with cats).

This makes DNA a remarkable dual-purpose molecule: it is both the functional driver of evolution and its archeological record. In the next section, we will try to abstract away from the biology and ask what properties make DNA so well suited to this role — because those properties, as we will see, are not unique to biological objects.

What makes a good marker of evolutionary history?

Before Darwin, evolutionary studies relied primarily on phenotype — researchers attempted to deduce evolutionary relationships from the appearance of species. But individuals are extraordinarily high-dimensional objects: every feature may evolve at a different rate, and many superficial similarities turn out to be the product of convergent evolution rather than shared ancestry. Two species can look alike simply because they adapted to the same environment, not because they are closely related. As Darwin himself argued, functional characteristics — what an organism does and how it works — are far more directly tied to evolutionary history than surface appearance alone.

This is precisely what makes DNA such an effective marker. It is a compressed and universal representation of the individual: vastly smaller than a full description of the organism yet capturing the core logic that shapes it, expressed in a common biochemical language that can be compared across wildly different species. It encodes functional identity rather than superficial similarity — the underlying architecture of the organism rather than its appearance. And crucially, at least for the right choice of genes, it evolves at a moderate and relatively steady rate — not so fast that the historical signal is lost in noise, and not so slow that no variation accumulates between populations.

These three properties:

(1) compression and universality (which implies a combinatorial encoding),
(2) a moderate evolutionary rate,
(3) and functional grounding,

are what makes DNA well suited to phylogenetic reconstruction. And if these are the key properties, then DNA need not be the only object that satisfies them. As we will explore in the next part, similar structures may exist far beyond biology.

The limits of tree-based models

The methodology described above still has important limitations, as the Bengal cat already illustrated. It assumes a strictly tree-like model of evolution, and therefore does not easily account for horizontal gene transfer, convergent evolution, or interbreeding between distinct lineages. In reality, evolutionary history is often more of a tangled web than a clean branching tree. It is important to keep in mind that phylogenetic trees are powerful approximations, but approximations nonetheless — and we will see this tension resurface in practice when we move beyond biology.

Part 3: Phylogenetics Beyond Biology

We have seen in the previous section that the properties of DNA which make it a particularly powerful tool for reconstructing evolutionary lineages may also be found in non-biological objects, opening the door to applying phylogenetic techniques beyond biology. This section presents several studies that do exactly this, focusing on cultural artifacts — objects created by humans that carry information about the culture of their creators and users — in order to trace their cultural evolution.

Cultural evolution

Just as biological species evolve under the pressure of their environment, cultural objects evolve under the pressure of the societies that produce and transmit them. An idea, a story, a tool, or a tradition does not remain static as it passes from person to person and generation to generation — it mutates, gets selected, and drifts, much like a genome. Some variations spread because they are more compelling, more useful, or more memorable; others disappear because they fail to resonate or survive transmission. This process, driven by human minds rather than biology, is what we call cultural evolution.

What makes cultural evolution particularly fascinating — and particularly amenable to phylogenetic analysis — is that it leaves traces. Cultural objects carry within them the marks of their history: a myth preserves echoes of the society that first told it, a tool reflects the constraints and knowledge of the civilization that built it. If we can identify the right markers of this history — the cultural equivalent of genes — we can reconstruct the lineages of these objects just as biologists reconstruct the tree of life.

Following subsections will demonstrate the criteria established in Part 2 are applicable to the reconstruction of cultural evolution. The thread connecting all the examples is: does this object have a compressed, universal representation that evolves in a traceable way?

Cultural artifacts and the extended phenotype (Dawkins)

Dawkins was a famous evolutionary biologist and his vision of evolution was never exclusive to biology. In The Selfish Gene (1976) [1], he introduced the concept of the meme — a unit of cultural transmission, analogous to the gene, that spreads from mind to mind through imitation and repetition. Ideas, melodies, fashions, and practices all evolve under selection pressure (what resonates gets copied and transmitted) and drift (stylistic variations that don’t affect cultural fitness accumulate over time). While this was a groundbreaking conceptual leap, Dawkins did not propose precise methods for inferring the phylogenetic history of memes — he did not address how to define the three properties we identified in Part 2: compression and universality, functional grounding, and a moderate evolutionary rate. That challenge was left for others to take up.

Myths and oral traditions

Julien d’Huy offers a concrete and compelling example of cultural phylogenetics in practice [2]. Myths — stories tied to religion, cosmology, and other foundational cultural concepts — evolve slowly enough to preserve historical signal across generations, satisfying the moderate evolutionary rate we identified as a key property in Part 2. D’Huy defines a “gene” as a high-level semantic feature of a myth: character names, key events, moral structure, and so on. He then measures the distribution of these features across cultures and uses them to build a phylogenetic tree. This approach produces a compressed and functional representation of each myth — capturing what the story does culturally rather than its surface details — with enough variance between cultures to be informative without being noisy. In other words, it satisfies all three properties we outlined in Part 2.

Let’s take a small example from A Cosmic Hunt in the Berber sky: a phylogenetic reconstruction of a Palaeolithic mythology, D’Huy 2013 [2]. Let’s consider 3 versions of a myth: Evenki 2, Basque 2 and Pausanias and “genes” (called mythems in D’Huy’s work):

Mythem	Evenki 2	Basque 2	Pausanias
There are at least 3 pursuers	1	1	0
Pursuers are dogs	0	1	0
The animal is dead when transformed into a constellation or in the sky	0	0	1

A “1” indicates that the myth includes this mythem — for instance, both Evenki 2 and Basque 2 feature at least 3 pursuers, while Pausanias does not. A “0” indicates its absence.

Computing the distance between these myths consists in counting the number of mythems on which they differ:

Distance	Evenki 2	Basque 2	Pausanias
Evenki 2	0	1	2
Basque 2	1	0	3
Pausanias	2	3	0

Evenki 2 and Basque 2 differ on only one mythem — whether the pursuers are dogs — making them close relatives in the tree. Pausanias, on the other hand, is distant from both. Applying the NJ algorithm to this distance matrix produces the following tree:

        ┌──────────────────── Pausanias
───┤
        └──┬──────────── Basque 2
                │
                └──────────── Evenki 2

This toy example extracted from the paper illustrates how mythems (akin to genes) can be used to reconstruct the evolutionary history of myths. The full figure from the paper is the following (which includes more mythem and myths): figure from D’Huy Julien, (2013) *A Cosmic Hunt in the Berber sky: a phylogenetic reconstruction of a Palaeolithic mythology [2]* The color of branches indicate geographic region of the myth - branches are consistent with what we know of first human migration patterns

The results are striking: the phylogenetic trees reconstructed from myth features seems to be consistent with known human migration patterns, reinforcing the idea that these semantic features can serve as an artificial DNA of cultural evolution. It is a beautiful case study in how phylogenetic thinking can travel far beyond biology.

That said, human migrations do not always follow a clean tree-like structure — cultures merge, split, and influence one another in complex ways. Just as the Bengal cat exposed the limits of tree-based models in biology, we might equally encounter a “Bengal myth” where the branching assumption breaks down. D’Huy often explains that this phenomenon appears to be rare in myths.

What is the “DNA” of an artifact?

Before moving to LLMs, let’s link d’Huy’s work back to the theoretical framework of Part 2 and make explicit what a phylogenetic analogy requires in practice.

Given a set of objects whose evolution we want to reconstruct, we need to define four things: what counts as an individual, what counts as a population, what the genes are, and what the alleles are. In d’Huy’s work, each myth is treated as an individual, with genes corresponding to high-level semantic features — the name and gender of the main character, key actions, moral outcomes — and alleles being the specific values those features take in each version. This is slightly different from the algorithm presented in Part 1, which works with populations and probability distributions. D’Huy’s approach can be seen as a special case where each myth is its own population of one: every feature has probability 1 for its observed value and 0 for all others.

For this to work, the set of genes chosen must be universal across all the myths being compared — just as comparing genes associated with mammalian biology makes sense across cat breeds but not across cats and birds. D’Huy’s semantic features satisfy this: broad enough to appear in all versions of a myth, yet specific enough to vary meaningfully between cultures. Heracles and Hercules, for instance, share the same essential structure — a hero completing a series of extraordinary trials — but differ in countless details. The high-level features capture the shared ancestry; the details carry the drift.

This also speaks to the question of selective pressure versus drift in cultural objects. Myths, unlike everyday stories or rumours, tend to preserve their core meaning across generations and cultures — their central themes are under a kind of cultural selective pressure, while peripheral details drift freely. This stability at the core and variability at the margins is precisely what creates the right amount of variance to build a meaningful phylogenetic tree.

In short, d’Huy’s method satisfies all three properties identified in Part 2: it produces a compressed and universal encoding of each myth through a small set of high-level features, those features capture functional and cultural meaning rather than superficial details, and they vary at a moderate rate — stable enough to preserve historical signal, variable enough to distinguish between lineages.

This framework highlights the relevance of using phylogenetic tools to study the evolution of cultural artefacts and it is interesting to think about the fact that it could go even beyond myths and be applied to other fields of cultural artefacts such as advanced technologies like generative AI. Indeed, these new generative models are created by humans and carry a lot of information about our culture. Additionally, modern models have a rich evolutionary history: they trained using training techniques, datasets and previous model checkpoints that could be particularly relevant to infer due to the extremely fast evolution of the field.

Part 4: LLM Phylogenetics

Having seen how biologists reconstruct the evolutionary history of living species, and how the same framework can be extended to cultural artifacts, let’s now ask how we might apply it to language models. This work has been published at ICLR conference in 2025 and is accessible here [6].

What is a LLM?

A language model is a large neural network trained to predict the next token given a context. This means that a language model is, mathematically speaking, a probability distribution P(token | context): for any sequence of tokens, it assigns a probability to every possible next token. This is not just a convenient description — it is the complete definition of a language model’s behavior. Two models that agree on P(token | context) for every possible input are, functionally, identical.

This gives us a natural and universal framework for comparing language models, since every LLM can be expressed in this form. However, compressing this identity into something tractable is a significant challenge: the space of all possible contexts is effectively infinite, making it impossible to evaluate P(token | context) exhaustively. A solution could potentially be to exploit their weights or activations but this makes their identity either non universal as the architecture of LLMs may change between families or non functional as permutations of a same weight set can lead to the same functionality. We encourage reader to check this work that attempted to use weights to build evolutionary history of LLMs but only within the same family successfully using weights as LLM identity [5]. Rather than trying to compress it further, we decided to work directly with this form — sampling a representative set of contexts and using the resulting distributions as our artificial DNA.

Then, LLMs are either trained from scratch (pretraining) using a large neural network that is randomly initialised and then trained to predict the next token on a large text corpus — this teaches the model language and a broad range of skills. However, pretraining alone is often not sufficient for casual conversation, like what we experience with ChatGPT. Models are therefore frequently finetuned: trained further from a pretrained checkpoint to acquire more specific capabilities. These further training stages can involve supervised learning (trained to predict the next token in a specialised dataset of human-AI interactions), Reinforcement Learning from Human Feedback (RLHF — where the model is trained to produce responses consistent with human expectations), or Reinforcement Learning from Verifiable Rewards (RLVR — where the model is trained to produce accurate, verifiable answers). The landscape of modern LLMs typically involves a pretrained model — often already finetuned several times by a large tech company — which is then finetuned by many others, hundreds of times per day, forming extremely large family trees. Beyond this vertical inheritance through finetuning, other mechanisms also shape model lineages: shared neural architectures, common training algorithms, and overlapping datasets all constitute forms of inheritance worth studying in their own right. This makes the landscape of modern LLMs a particularly rich and timely subject for tools borrowed from genetics.

There is one remaining obstacle to universality: different language models use different tokenizers, meaning that the same text is split into different tokens depending on the model. For example, a LLM may not have a token “strawberry” but only one for “straw” and another for “berry” making it particularly hard to compare two LLMs with different tokenizer as the set of each function is not the same. A direct comparison of P(token | context) across models would therefore be comparing non comparable quantities. To sidestep this, we work instead with P(next 4 characters | context), converting token probabilities into character-level probabilities. This makes the representation truly universal — comparable across all language models regardless of their tokenizer.

LLMs as populations of text

Now that we have established the formal identity of an LLM, let’s develop the population analogy more carefully.

In Part 1, a population was defined as a probability distribution over traits: for each gene, it tells you how likely you are to sample an individual with a given allele. The key word here is sample — a population is something you draw individuals from. This is precisely how a LLM behaves. Generating text from a language model — token by token, each conditioned on what came before — is equivalent to sampling an individual from a population. The LLM defines the probability of each possible text, and generating from it draws one individual from that distribution.

More formally, a LLM being defined as P(token | context) naturally induces a probability distribution over texts:

$P(t_0 \ldots t_n) = P(t_0 | \varepsilon) \times P(t_1 | t_0) \times \ldots \times P(t_n | t_0 \ldots t_{n-1})$

This function is by essence a density over the space of all possible texts — the population the LLM represents.

To make this intuitive: a capable LLM represents a population enriched with texts like “Question: What is 2+2? Answer: 4”, while texts like “Question: What is 2+2? Answer: 3” have been effectively eliminated through finetuning. A less capable model would assign more mass to incorrect answers. Finetuning, in this framework, is analogous to selective pressure in biology: it shifts the population toward individuals (texts) that are more fit, and away from those that are not. Supervised finetuning does it with a database of “good individuals” to reproduce further while RLHF does it more subtly through a reward function that rates individuals and select the more fit.

To illustrate previous example better, LLMs are rarely used to generate a single token but rather a long completion to a given text. This is done by iterating the next token prediction autoregressively on the previously generated text:

$P(\text{ice cream}|\text{I like}) = P(\text{ice}|\text{I like})\times P(\text{cream}|\text{I like ice})$

Therefore P(completion | context) is just a rewriting of P(token | context) in a less compact form but they are equivalent.

Seen through this probably more understandable angle, generating a continuation given a context corresponds to sampling from a subpopulation:

$P(\text{continuation} | \text{context}) = \frac{P(\text{context} + \text{continuation})}{P(\text{context})}$

The proportion of a LLM’s population that contains a given context followed by a given continuation reflects how strongly that model associates the two — which is a more elegant way to write the identity of a LLM in order to understand what finetuning does to a LLM: selecting the completions that fit the training procedure given a context.

What is fascinating about this formalism is that all LLMs, irrespective of their tokenizer, training, or architecture, represent a population of text. The texts themselves are the same across all populations — only the probability each model assigns to them differs, and it is precisely this difference that distinguishes a capable LLM from a simpler one. This highlights the universality of the formalism and is what makes it so well suited to comparison across models.

Mathematically inclined readers will note that for $P(t)$ to be a proper probability distribution, texts must either be infinite or terminate with an end-of-sequence token — but we won’t dive into those details here. Additionally, P(token | context) defines P(completion | context) but not the opposite - once more this does not impact the message of this blog.

Genes and alleles in LLMs

Comparing P(token | context) directly across LLMs is extremely challenging in practice: the space of possible tokens and contexts is so vast that exhaustively evaluating it for two models would be computationally prohibitive. This is why, much like in genetics, we do not compare DNA directly but instead work with genes and alleles — a more compact and interpretable abstraction. The choice of how to define genes and alleles in the context of LLMs is not trivial, and several approaches are possible.

One option, inspired by d’Huy’s work, would be to identify high-level semantic features of generated text: defining genes as meaningful structural elements of a response and alleles as the specific values they take. This would produce a functional and interpretable comparison between models. However, d’Huy’s approach requires significant manual effort and was feasible precisely because the number of myths he studied remained small. In the current landscape of LLMs — where more than 300 new text-generation models are published every day on the HuggingFace hub — this kind of hand-crafted analysis is simply not scalable.

A more explicit automatic approach would be to let each model generate full continuations from a set of contexts, and then compare those continuations using an embedding model. Here, the continuation plays the role of the allele and the context plays the role of the gene. This method has been explored recently with very promising results using long text generations [8]. Another proposed approach estimates a KL divergence between models based on the probability they each assign to a set of texts — going back to the idea of LLMs as populations of texts — also with strong results on long texts [7].

PhyloLM was designed with a different, more lightweight approach: treating the next token (4 characters in practice) generated by the model as the allele for a given context gene. This is directly grounded in the definition of a LLM — P(token | context) — and is cheap to evaluate. It captures the evolutionary trace we are looking for: as a model is finetuned, the probability it assigns to each token given a context shifts, reflecting the selective pressure applied to its population of texts. In the next section we will see how this quantity is used in practice to compare models and build a phylogenetic tree of LLMs.

Selective pressure and drift in LLMs

We have discussed several times how finetuning corresponds to selective pressure in the biological framework — let’s now develop this further. As introduced in section about what is a LLM, once pretrained, a model represents a population that will be shaped and refined by each successive finetuning stage, each one applying a selective pressure on the population of texts the model represents.

As in biology, however, not all aspects of the model evolve under this selective pressure. During RLHF or RLVR, if the model is rewarded for giving the correct answer to a question, the format of that answer may not matter much. Whether the model responds to “What is 2+2?” with “4”, “The answer is 4”, or “The answer is 4.” — all of these will be rewarded equally as long as the answer is correct. This means the stylistic dimension of how LLMs answer questions behaves like a neutral mutation — a form of genetic drift. Anyone who has used several different LLMs will have noticed this: models do not feel the same, even when they give the same answer. They phrase things differently, have different rhythms, different default tones. If the model was never explicitly trained on these stylistic choices, they are part of its drift.

This also reinforces the choice of using next-token prediction as our allele rather than full completions compared via an embedding model. An embedding model tends to focus on the semantics of what is generated — closer to the selective pressure axis — while next-token prediction captures finer-grained stylistic and syntactic variation, potentially closer to the drift axis where the richest evolutionary signal lives.

It is worth noting that the analogy with biological drift has its conceptual limitations. In biology, neutral mutations accumulate at a relatively steady and quantifiable rate. In LLMs, training on a selective axis will inevitably perturb the probability of tokens unrelated to that axis (as training influences neurons that can be used in other unrelated tasks) — but how much, and in which directions, is much harder to quantify. The LLM drift axis is conceptually analogous to its biological counterpart, but its precise dynamics remain an open question that warrants further study. We hypothesised this drift is present in training LLMs but maybe with a lower rate than in biology.

This tension between selective pressure and drift is also practically important when choosing which genes to use — since, as we established in Part 2, we want genes that show moderate variance at the timescale of the study. For PhyloLM, we wanted to map the entire landscape of modern LLMs across roughly three years of development — a period largely defined by a drive to improve reasoning, mathematics, and coding abilities. Using contexts closely aligned with this selective pressure axis, such as well-known benchmark problems, risked showing too little variance as most modern models have been explicitly trained on these potentially capturing the same exact behaviour of these questions. On the other hand, using completely unrelated contexts such as poetry risked too much variance. We therefore chose a middle ground: short code and math snippets drawn from sources unlikely to appear in standard training benchmarks, cut at random mid-sentence. For example here are 4 “genes”:

# In observing a Tetrahedron…

# How to prove that $C={x: Ax\le

let sortedArr = arr.sor

if (monthnum3 == 6)

These contexts are specific enough to constrain the space of plausible 4-character completions, while open enough that different LLMs will complete them in meaningfully different ways — sitting comfortably in the moderate-variance sweet spot we are looking for.

PhyloLM: results and interpretation

The PhyloLM algorithm is described in details in this paper. In short, it estimates P(token | context) across a set of language models by sampling 32 completions per gene context. Sampling multiple completions rather than reading token probabilities directly makes the method compatible with proprietary models that do not expose their full probability distributions — and working with the first 4 characters rather than the first token, as discussed earlier, ensures universality across tokenizers.

From these samples, a population function is estimated for each model, and the Nei similarity score is computed between every pair:

$S(P_1,P_2) = \frac{\sum_{g\in G}\sum_{a\in A_g}P_1(a|g)P_2(a|g)}{\sqrt{(\sum_{g\in G}\sum_{a\in A_g}P_1(a|g)^2)(\sum_{g\in G}\sum_{a\in A_g}P_2(a|g)^2)}} $

where $P_1$ and $P_2$ are the population functions of two models, $G$ is the set of gene contexts, and $A_g$ the set of possible 4-character completions for gene $g$. The similarity matrix $S$ is bounded in $[0,1]$. A distance matrix is then derived from it:

$D(P_1,P_2) = -\log(S(P_1,P_2))$

and a phylogenetic tree is computed from this distance matrix using the Neighbour-Joining algorithm described in Part 1.

Illustration of the PhyloLM algorithm

This method makes it possible to accurately reconstruct the evolutionary history of LLM families. To illustrate this, we ran an experiment on a family of models for which the ground truth evolutionary tree is known, and used PhyloLM to reconstruct it using only their artificial genetic material — with no prior knowledge of their training history. The result is striking: PhyloLM reconstructs the tree with near-perfect accuracy, as shown below. PhyloLM tree reconstruction results

The left panel shows the ground truth and the right panel shows the reconstruction — the two are nearly identical, validating both the analogy and the method.

That said, the reconstructed tree does not include common ancestors. This is standard in biology, where the NJ algorithm is designed to reconstruct relationships between living species whose common ancestors are long extinct and therefore unobservable. In LLMs, however, we have access to all intermediate models — the “ancestors” are not extinct, they are publicly available. This makes the inability to recover common ancestors a meaningful limitation of the current method, and an interesting direction for future work as well as an interesting conceptual observation.

It is also possible to plot a more general dendrogram not taking into account the assumptions of not having common ancestors but also by including several families of models all at once:

Full dendrogram containing hundreds of LLMs.

This dendrogram is a very nice visualization that, first, PhyloLM is capable of clustering families of models (models finetuned from the same pretrained model show similar behavior) but also that some of these families are similar (like OPT, Pythia and GPT-3) that were likely pretrained on similar training sets. Nonetheless, it is important to remind that the asumptions for using the NJ algorithm are not respected in this context and therefore this cannot be interpreted as an evolutionary tree but rather as a distance visualization tree.

Limitations and open questions

Now that we have seen how PhyloLM works and why the analogy holds, let’s take a step back to recap its assumptions and discuss the limitations and open avenues for future work.

We established that P(token | context) satisfies the three properties identified in Part 2 for a good evolutionary marker: it is universal across all LLMs (if not particularly compressed), it can be made to exhibit moderate variance at the relevant timescale through careful gene selection, and it is the functional grounding of language models by definition. These properties make it well suited to phylogenetic reconstruction — but only under the assumptions that the method relies on.

The most significant of these is the tree-like model of evolution. As we noted in biology, real evolutionary history is closer to a graph than a tree: horizontal gene transfer, convergent evolution, and interbreeding all introduce connections that a tree cannot capture. LLMs face the same issue. Training data shared across model families, distillation — where one model is trained to imitate another — and other forms of cross-lineage influence are all forms of horizontal evolution that the current method does not account for.

There is also a structural difference between biological and artificial evolution that pushes beyond the tree assumption in a deeper way. In biology, it is often considered that all known life shares a single common ancestor: the tree has one root. In LLMs, this concept is more complex. If we consider only vertical inheritance through finetuning, multiple independent roots exist — GPT, Mistral, Llama, and others were each pretrained from scratch by different actors. If we adopt a broader definition of inheritance — encompassing shared neural architectures, training algorithms, and datasets — then a single root becomes more conceivable, perhaps traced back to the original Transformer architecture, but the resulting tree structure becomes far more tangled.

This ambiguity also raises a deeper question about what the tree is actually capturing. Consider two pretrained models A and B, the former finetuned into models C and D with respectively method 1 and 2 and the later in E using method 2. Should C and D be close because they share an ancestor, or should D and E be close because they were finetuned both with method 2 ? PhyloLM captures behavioral similarity without distinguishing between these sources of relatedness, making the interpretation of the resulting tree sometimes less trivial than in biology due to all these horizontal transfers in the evolutionary history of LLMs.

More fundamentally, the flow of genetic information in LLMs is far less constrained than in nature. In biology, populations speciate precisely because contact is lost — genes cannot flow freely between isolated groups. In the LLM ecosystem, training practices, architectures, and datasets circulate freely through scientific papers, open-source releases, and shared benchmarks. This sometimes makes LLM evolution look less like a branching tree and more like a single large population that periodically speciates into new niches (generative AI, chatbots, coding assistants, etc.) as new scientific directions emerge — closer, perhaps, to the evolutionary tree of science than to the traditional tree of life observed in biology.

These observations point to important directions for future work. Current phylogenetic algorithms, designed for the slower and more isolated dynamics of biological evolution, are not well suited to this rapid and highly horizontal form of inheritance. New methods will be needed to handle multiple roots, mixed inheritance signals, and the graph-like structure that better describes how LLMs relate to one another. Additionally, as noted earlier, the NJ algorithm cannot recover common ancestors — a limitation that matters more in LLMs than in biology, since intermediate models remain available and observable. Developing methods that can reconstruct internal nodes, not just leaf relationships, would be a natural and valuable extension.

Ultimately, these limitations are not just obstacles but should be seen as invitations to rethink the field of genetics. They pave the way for a larger sense of genetics, not anchored to biological DNA sequences, but built around general atoms of evolution shared across biological and artificial life forms.

_{Cite PhyloLM}

@inproceedings{ICLR2025_a2e28663,
 author = {Yax, Nicolas and Oudeyer, Pierre-Yves and Palminteri, Stefano},
 booktitle = {International Conference on Learning Representations},
 editor = {Y. Yue and A. Garg and N. Peng and F. Sha and R. Yu},
 pages = {64792--64844},
 title = {PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks},
 url = {https://proceedings.iclr.cc/paper_files/paper/2025/file/a2e28663712d5a3429a98918c3058f7b-Paper-Conference.pdf},
 volume = {2025},
 year = {2025}
}

_References

[1] Dawkins, R. (2006). The Selfish Gene. Oxford University Press.
[2] d'Huy, J. (2013). A Cosmic Hunt in the Berber sky: a phylogenetic reconstruction of a Palaeolithic mythology. Les Cahiers de l'AARS, 15, pp.93–106.
[3] Takezaki, N. & Nei, M. (1996). Genetic Distances and Reconstruction of Phylogenetic Trees From Microsatellite DNA. Genetics, 144(1):389–399.
[4] Saitou, N. & Nei, M. (1987). The neighbor-joining method: a new method for reconstructing phylogenetic trees. Molecular Biology and Evolution, 4(4):406–425.
[5] Howitz, E. & Kurer, N. & Kahana, J. & Amar, L. & Hoshen, Y. (2025). We Should Chart an Atlas of All the World's Models. ICLR 2025.
[6] Yax, N. & Oudeyer, PY. & Palminteri, S. (2025). PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks. ICLR 2025.
[7] Momose, O. & Yamagiwa, H. & Takase, Y. & Shimodaira, H. (2025). Mapping 1,000+ Language Models via the Log-Likelihood Vector. ACL 2025.
[8] Wu, Z. & Zhao, H. & Wang, Z. & Guo, J. & Wang, Q. & He, B. (2026). LLM DNA: Tracing Model Evolution via Functional Representations. ICLR 2026.

Can AIs understand our world? Functionally grounding LLMs in interactive environments.

Thu, 06 Feb 2025 00:00:00 +0000

ChatGPT, GPT-4, Mistral, Gemini… In recent months, a flurry of “language models,” mostly developed by major companies such as Google, Facebook, and OpenAI, have emerged. These intriguing tools, now widely used by the general public (who hasn’t heard of ChatGPT at a family gathering?), all operate on the same principle: software capable of responding to text-based messages from users.

Whether it is answering questions about historical facts, telling jokes, or generating cover letters, these tools initially appear impressive in their capabilities. However, upon closer inspection, certain limitations become evident. Notably, these systems sometimes produce false or entirely fabricated information. This makes distinguishing truth from falsehood particularly challenging since the model itself cannot discern the difference and presents all outputs with equal confidence.

Some inaccuracies, however, immediately raise suspicion, especially when they seem completely disconnected from the world as we know it. Consider, for instance, this absurd explanation from ChatGPT: an anaconda cannot “fit, size-wise, in a shopping mall” because its length would pose a problem given the building’s ceiling height.

_{Despite their impressive capabilities, LLMs still fail at simple tasks by providing answers that appear disconnected from our world (such as the anaconda not fitting a shopping mall). Illustration generated with FLUX.1[dev]}

Why do models miss the obvious? How can something so apparent to us escape a language model entirely? To understand this, we must delve into the nature of these systems. Language models are based on a mathematical function with an enormous number of parameters—over 100 billion in today’s models. These parameters are adjusted to produce the desired output for a given input. Using mathematical methods known as Machine Learning, these parameters are automatically tuned to match inputs and outputs in a training dataset (a set of input-output pairs). In practical terms, each input word is assigned a representation vector (a point in a multidimensional real-number space), and the output is a probability distribution over possible next words (within a predefined vocabulary). Training a language model involves exposing it to vast amounts of text and tuning its parameters so that it assigns the highest probability to the correct next word in each context. To generate text, the model starts with an initial phrase, selects the most probable next word, and iterates this process, appending each newly chosen word to the input.

This training principle predates the recent explosion of language models. Although employed for decades, earlier iterations produced encouraging but far less impressive results compared to today. What changed? First, the mathematical function that must be tuned evolved. Until 2017, language models primarily relied on recurrent neural networks (RNNs), which process one word at a time and use a memory matrix to retain information across words. In 2017, a new model called the transformer was introduced. Unlike RNNs, transformers consider all words in the context simultaneously, using a mechanism called attention to relate representation vectors of different words. Moreover, transformers can perform many operations in parallel, enabling the training of models with tens of billions of parameters and vast datasets.

Because language models produce a probability distribution for the next word, they can generate text but also calculate the likelihood of one phrase following another. For instance, if GPT-3 is asked about its favorite color, the next word probabilities reflect the preferences expressed by humans during training.

The symbol grounding problem

This passive learning gives models some knowledge about our world. However, this knowledge is purely “bookish”. It doesn’t stem from direct experience of reality. Yet, it is this direct experience that enables us to intuitively understand whether, say, an anaconda can navigate the aisles of a supermarket.

How can AI acquire this capability? And, more fundamentally, is this even desirable? Beyond attempting to mimic natural processes, grounding words in the real world is crucial if we want models to be genuinely useful beyond generating jokes, summarizing texts, or structuring documents. For example, we need to share a common foundation of knowledge with these models so that the words we use to communicate with them carry the same meaning or refer to the same concepts.

Take the example of asking a robot to “pick up an apple.” The robot must connect the word “apple” to the physical reality of an apple, accounting for variations in its color, size, shape, and taste.

Language models extract factual knowledge from their training data (e.g., history books, recipes) and understand object relationships (e.g., knives cut, spoons don’t). This allows them to predict common interactions, forming a limited model of the world. Humans learn similarly, especially children, with stories aiding understanding. However, human learning also involves interaction within a sociocultural context, where words are tied to physical and social realities. Unlike humans, language models lack this grounding in the real world.

This issue of grounding is well-known in research. It was formalized in 1990 as the symbol grounding problem by cognitive scientist Steven Harnad, who posed the following question: Can a system of rules defining possible combinations of symbols (such as words or hieroglyphs) inherently contain meaning and refer to objects or concepts outside this system? Intuitively, the symbol grounding problem examines how symbols—such as the words we use—can be associated with the world around us to carry meaning when interpreted in context. Psychologists and linguists have extensively studied grounding, and it has also inspired research in Machine Learning. Much of this research has focused on associating object names with their visual representations. This has led to numerous image generators capable of producing visuals from textual descriptions. Some of these advancements have been applied to modern language models. For example, models like GPT-4 can now accept both text and images as inputs. Yet, the broader grounding of these models in the physical world remains unsolved. The current mechanisms still fail to capture the meaning of certain concepts, such as physical properties, as evidenced by ChatGPT’s anaconda scenario.

What would it take for a language model to grasp the fundamental concept of gravity, a cornerstone of our reality? Could such grounding enable a robot, conversing like ChatGPT, to perform practical tasks in the physical world?

_{Functional grounding implies active learning mechanisms: one must interact with its environment through actions and perception to functionally ground symbols. Illustration generated with FLUX.1[dev].}

Answering these questions requires exploring another type of grounding: functional grounding. This approach focuses on the symbols used to act in a given environment and predict the outcomes of actions, enabling problem-solving within that environment. By solving problems, the model can ground the physical dynamics or rules of its environment. In July 2023, we introduced “Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning” (aka GLAM). In this paper, we proposed the first approach to functionally ground LLMs.

Towards functional grounding

This way of grounding reality in our representations, particularly the actions that change our world, is something we naturally rely on, especially when using our inner speech (endophasia) to list actions needed to achieve a goal. For instance, when cooking, it is common to mentally plan the next steps to create the desired dish. Symbols (such as actions like “cut,” “cook,” etc.) are used to act on our environment and to predict its state after the interaction. The grounding here is specific to each environment encountered—“advance” for example, does not mean exactly the same in the context of a chess game and in our everyday world.

How can this mechanism be applied to an AI system? How can we link a language model to an environment, whether it is the physical world or a computer simulation, in a way that connects symbols, actions, and the outcomes of these actions? For this initial study of functional grounding, we needed to create a dedicated environment that would allow us to isolate functional grounding from other forms of grounding and study its unique properties.

We required an environment in which an agent could act and where its actions would affect that environment. Moreover, the agent needed to decide which action to take based on the outcomes of its previous actions. In practice, our agent is nothing other than a language model, which serves as both the body (capable of action) and the brain (capable of decision-making).

The concept of an agent is fundamental and highly prevalent in the field of Artificial Intelligence. We adopt the definition provided by Stuart Russell and Peter Norvig: an agent is “any entity that can be considered as perceiving its environment through sensors and acting upon that environment via effectors”.

_{We introduce BabyAI-Text as a testbed for functional grounding. It extends the classic BabyAI test bed initially designed for other forms of grounding.}

BabyAI-Text, an experimental testbed for functional grounding

We proposed to use an interactive textual game called BabyAI-Text. While more complex environments, such as those involving visual perception (e.g., a robot with a camera interacting with objects in a kitchen), are enticing, they involve additional forms of grounding (like linking an object’s name to its various visual facets). Instead, we opted for a simulation simple enough for its elements to be directly described in text, yet rich enough to explore different properties of functional grounding. Our system resembles early video games, such as [Zork], a turn-based text adventure where a character or agent receives a description of what it “sees” and can send text-based commands (e.g., “attack the dragon with the sword”) back to the environment. When the game begins, the player is given a goal (e.g., “defeat the dungeon monster”) along with an initial description of their surroundings. Achieving the goal requires performing various actions, more or fewer depending on the chosen strategy. Each turn, the player types the desired action the agent must perform and receives an updated description of their perception.

In BabyAI-Text, the agent operates in a room where it can move using three commands: “move forward,” “turn right,” and “turn left.” The room contains objects the agent can pick up and move. The agent receives a goal such as “Place the red ball next to the blue box.” On each turn, it can only see part of the room and must choose an action from six possible options (three for movement and three for object interaction). Once the goal is reached or the turn limit is reached, the room’s content is reset, and a new, randomly assigned goal (possibly composed of multiple simpler goals) is given.

GLAM, functionally grounding through online interactions

In our experiments, the agent is controlled by the large language model [Flan-T5] (with 780 million parameters, developed by Google in 2022). The “brain” of the agent was therefore initially trained by its creators to predict the right next token given text inputs drawn from millions of documents. It must now learn to judiciously use the vocabulary associated with its interactions in BabyAI-Text. For example, when the model chooses to use the command “move forward,” it must associate the word with what moving forward entails in the environment (e.g., the perception changes, objects in front of the agent move closer, and if there is an object already in front, it remains in place). We call the method enabling this learning GLAM (Grounded LAnguage Model).

_{In GLAM, we assemble the textual perception given by BabyAI-Text into a prompt that is given to the LLM to functionally ground. This LLM then selects an action to perform by compyting the probability of each possible action in BabyAI-Text.}

To perform functional grounding, instead of direct agent-environment interaction, we could have opted to show the language model explanations (in sentence form) of the environment’s dynamics and train it to reproduce those explanations (i.e., predict the next word in the provided explanation). However, it is not always straightforward to encode the physics of an environment in sentences. For instance, explaining that movement in an environment changes what is subsequently perceived is challenging. This concept seems easier to grasp through direct interaction with the environment and learning from the results of those interactions. Furthermore, learning to recite explanations does not necessarily mean the model has anchored the dynamics or can use them to solve problems in the environment. There is no guarantee.

Another possibility would be to provide the model with input-output pairs, where the input contains the goal to solve and the agent’s observation, and the output specifies the chosen action. But where would these examples come from? Should we ask a human to interact with the environment for thousands of turns? Or create another artificial agent to explore the environment and provide the language model with these examples? A major drawback of this method is that passively inferring causal effects from observing another agent’s actions can introduce confounding factors—information that influences both the agent’s decisions and the results (the subsequent observations). For instance, an autonomous car learning to drive by observing humans might infer that braking makes pedestrians appear, as it often observes pedestrians wanting to cross when the car brakes, potentially leading the car to avoid braking altogether.

This is why, in our study of functional grounding, the proposed agent generates (or rather assembles in a predefined way, via a “prompt”) a text containing the goal, the observation description, the list of possible actions, and a question asking what action to take. This text serves as input for the language model, which is expected to provide the continuation: the next action.

It is essential to remember that a language model is a mathematical function used to compute the probability of a complete sentence following a given text. This principle is central to GLAM. Instead of generating the most likely text sequence following the input (which might not correspond to an action possible in the environment), GLAM uses the language model to directly calculate the probability of each action given the input. GLAM evaluates all options (“move forward,” “turn right,” etc.) appended to the prompt ending with the question and selects the action with the highest probability according to the language model.

The agent, equipped with the prompt assembling information returned by the textual environment, effectively has both a brain (the language model) and a body. How does it update its brain’s knowledge based on its body’s actions? In the GLAM approach, we use Reinforcement Learning. In this subfield of Machine Learning, an agent tests various strategies (associating actions with observations) to maximize a reward provided by the environment after a series of interactions. The strategy parameters are gradually adjusted without experimenter intervention to achieve the best possible reward. This method enabled DeepMind to develop [AlphaGo], the Go-playing agent that defeated the world champion, and OpenAI to train an agent to solve a Rubik’s Cube using a robotic hand [2].

Is this enough to functionally ground a language model on its environment? We conducted a total of 1.5 million interactions between Flan-T5 and BabyAI-Text to train the agent to solve various types of goals in different rooms. Tested on 1,000 new room-goal combinations unseen during training, the grounded version of Flan-T5 (named GFlan-T5) successfully completed 89% of tasks requiring it to reach or retrieve an object, compared to only 11% for Flan-T5 without any grounding. Remarkably, GFlan-T5 maintained an 87% success rate even when the objects in the room were entirely new (training tasks only involved balls, boxes, and keys, whereas tests included chairs, tables, and cars). When these new objects were assigned invented names (“axfe,” “xolo,” “dax”), GFlan-T5 still succeeded in 88% of tasks. These results demonstrate how GLAM’s functional grounding improved Flan-T5’s handling of vocabulary tied to environmental dynamics (e.g., “move forward,” “turn”) without disrupting unrelated vocabulary, such as object names.

Finally, we compared GLAM’s active grounding approach to a passive grounding method. We created an expert agent specifically for BabyAI-Text with access to more information than the agent presented above. It could solve all tasks using a hand-designed, optimal strategy (which does not generalize to other environments). Using this expert, we recorded as many “turns” as GFlan-T5 explored. Each turn’s goal, observation, and possible actions were formatted as a prompt, and the same Flan-T5 model was trained to predict the expert’s chosen actions (instead of discovering on its own a strategy using GLAM). This created a passive functional grounding, where the language model learns to choose the correct action without interacting with the environment.

Results showed that passive learning was less effective than GLAM’s active approach, even when provided with examples of the best actions for each turn. Moreover, when room objects changed, the passively trained agent performed significantly worse. This gap is explained by the limited exposure of passive grounding to diverse examples. For instance, GFlan-T5 learns to correct errors (e.g., retracing steps), while passive agents lack such adaptability. Additionally, passive grounding may introduce confounding factors that are challenging for the model to untangle [3]. Lastly, the active Reinforcement Learning used by GLAM trains the model to select actions aimed at achieving the final goal, emphasizing long-term strategy acquisition.

Towards grounded LLMs in robots?

With the introduction of a novel approach to the famous symbol grounding problem, as well as a pioneering method for functionally grounding a language model, a new path opens up toward future AI systems more deeply embedded in our world. This is, in fact, only an initial step, currently tested on relatively small models in a limited framework. However, this work could inspire other research teams to better understand and address the functional grounding of larger language models.

Another important question is how such grounding can be implemented in our world. Recent research [4][5] has integrated language models into robots interacting with the physical world, using various Machine Learning models to describe what the robot’s camera perceives. These studies have demonstrated that, similar to the work with GLAM, the language models can select actions to solve tasks assigned to the robot. However, they do not involve functional grounding to align the language model with the environment or to correct its decision-making in case of errors. Such a framework would require studying not only the functional grounding involved in decision-making but also the grounding of the mechanisms that transform camera images into textual descriptions and the grounding of the process that converts language model-described actions into sequences of electrical impulses sent to the robot’s motors.

Finally, what about the impact functional grounding has on the language model when it is later used to generate text and answer questions? Is this the best way to imbue its responses with common sense? These are also open questions this first study of functional grounding raises.. For instance, would functionally grounding a language model like the one powering ChatGPT in an environment where it must place objects of the correct size into a given container suffice to ensure the conversational agent no longer makes mistakes like the “anaconda in the shopping mall?

_{Cite GLAM}

@InProceedings{pmlr-v202-carta23a,
  title = {Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning},
  author = {Carta, Thomas and Romac, Cl\'{e}ment and Wolf, Thomas and Lamprier, Sylvain and Sigaud, Olivier and Oudeyer, Pierre-Yves},
  booktitle = {Proceedings of the 40th International Conference on Machine Learning},
  pages = {3676--3713},
  year = {2023},
  editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = {202},
  series = {Proceedings of Machine Learning Research},
  month = {23--29 Jul},
  publisher = {PMLR},
  pdf = {https://proceedings.mlr.press/v202/carta23a/carta23a.pdf},
  url = {https://proceedings.mlr.press/v202/carta23a.html},
}

_References

Carta, T. et al. (2023). Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning. Proceedings of the 40th International Conference on Machine Learning
Bruner, J. (1990). Acts of meaning. Harvard University Press.
OpenAI et al. (2019). Solving Rubik's Cube with a Robot Hand.
Gasse, M., Grasset, D., Gaudron, G., & Oudeyer, P. (2023). Using Confounded Data in Latent Model-Based Reinforcement Learning. Trans. Mach. Learn. Res., 2023.
Ahn, Michael et al. (2022). Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. Conference on Robot Learning.
Huang, Wenlong et al. (2022). Inner Monologue: Embodied Reasoning through Planning with Language Models.

An ecological approach to Artificial Intelligence

Wed, 09 Aug 2023 00:00:00 +0000

Designing artificial conversational agents to train children's curiosity during learning, a proof of concept through the Kids Ask project

Tue, 11 Oct 2022 00:00:00 +0000

[EAET 2022 Conference talk] ● [IJHCS 2022 paper] ● [Blog article PDF]

This blog post presents our recent IJHCS paper Conversational agents for fostering curiosity-driven learning in children.

Curiosity : a key factor for enhancing learning experiences and outcomes

Curiosity is defined as “a desire to know, see or experience that motivates exploratory behavior oriented towards the acquisition of new information” [2]. It is an important factor that can improve children’s ability to take charge of complex activities [3], to engage in learning [4], or to memorize new information better [5]. Indeed, our brain learns and memorizes the information better when we seek it and find it on our own, by asking questions or exploring the environment for example, and not by passively receiving it [6]. Given its importance for learning, several recent studies have focused on the study of curiosity in the classrooms, with the goal of designing pedagogical interventions that can foster this ability in young children. Their obseravtions showed that it is, indeed, a malleable skill [7] that can be elicited by verbal and nonverbal cues [8,9,10], by promoting comfort with uncertainty and encouraging questioning and exploration to resolve it. However, and despite different efforts, recent reports show that today’s educational content does very little to encourage children to be curious, generally reducing them to one correct answer and not allowing them enough space to ask questions and explore, the main expressions of epistemic curiosity.

Curiosity during learning : the major brakes

Questioning behaviors, which are the primary expressions of epistemic curiosity in children, are almost absent in today’s classrooms. Several factors may be responsible for this; in particular, the so-called “knowledge illusion”: children’s tendency to overestimate their knowledge levels and not be aware of the information they might be missing - this is the notion of “knowledge gaps” [16]. This problem stems mainly from their lack of reflection on their own learning and their inability to assess their own knowledge levels.

This capacity of self-assessment and the awareness of knowledge gaps are very important factors to stimulate the curiosity of the individual. Indeed, when we realize that we do not have a piece of information or that our knowledge does not correspond to the one necessary to understand the environment, a knowledge objective to acquire will appear. This objective is usually a motivator for a curiosity-driven behavior -such as generating a question-, which is initiated in order to compensate for the identified “knowledge gap” and acquire the missing knowledge [17]. Other factors that may prevent children from asking questions in class are their inability to formulate syntactically correct questions [11], or their fear of judgments from peers or their teachers [12].

To remedy these problems, several research studies encourage teachers to set up specific training sessions to familiarize children with the notions of self-evaluation and questioning and to help them feel more comfortable with their uncertainties. However, such workshops can be difficult to create as they require teachers to find specific teaching materials for each knowledge component and activity, which can be a particularly time-consuming task [1,10].

The “Kids Ask” project

In this context, and in order to support teachers in this task, our “Kids Ask” project leverages new technologies and proposes a curiosity training based on the awareness of “knowledge gaps”; paying particular attention to self-assessment and how to avoid the “knowledge illusion” trap. In particular, we implement conversational agents capable of stimulating questioning and exploration motivated by the desire to compensate for specific missing information. Our work is notably motivated by the positive effects that this type of technologies has shown on children’s learning strategies, their divergent thinking, as well as for the construction of good social relationships with them [13].

Design

“Kids Ask is an web-based educational platform prototype, that involves interaction with a conversational agent. The platform is designed to help children identify their uncertainties, or “knowledge gaps”, generate questions to compensate for these gaps and, finally, use their questioning skills to explore and acquire new knowledge autonomously by following their curiosity.

To meet this objective, “kids Ask” offers three work spaces:

Curiosity stimulation Space: The agents in this space offer quizzes on different topics and try to make students aware of gaps in their knowledge by encouraging them to report confidence levels in the answers they give. We use this strategy to probe students’ meta-cognition and attempt to arouse their curiosity by making them aware of gaps in their knowledge. Indeed, these strategies allow children to think more deeply about what they know and do not know and thus be able to defer more uncertainties [14]-the first step to adopting curious behaviors.
Curiosity elaboration space: In a second step, our agents aim to help children go one step further than the identification of uncertainty: we want to train them to know how to express these uncertainties and to be able to pursue them by asking the appropriate questions.
To do this, the agents propose reading-comprehension tasks and specific cues for formulating questions that are said to be “divergent” from the texts. These are questions that bring up new information about the text and ask students to make hypotheses, connections between different ideas, etc. The agents help the students in this exercise by proposing precise linguistic and semantic clues that lead to this divergent thinking. The idea here is to propose “knowledge gaps” to the students and to encourage them to pursue them by formulating high-level questions. By doing this exercise, children could thus acquire new information that they would have looked for on their own, and which is important for a better understanding of the text. We choose to focus on training divergent questioning because it is an ability that induces more curious thinking and involves the use of more cognitive processes [15].
Curiosity maintenance space: In order to help children stay in curious states, our agents encourage them to mobilize their questioning abilities to explore educational resources available in the platform and perform autonomous, organized, and personalized investigations. The purpose of this space is to train children to take time to evaluate the new information they acquire and ask themselves if they are satisfied with it or if it raises new questions in them. Such exercises help avoid the trap of the “illusion of knowledge” and the “premature” interruption of learning cycles.

_{Figure 1. Illustration of the 'Kids Ask' platform features, with relation to our curiosity model}

Technical implementation

The interface was programmed in JavaScript using the REACT library and was connected to a RESTful API to publish and retrieve the interaction data. The behavior of the agent in terms of selection of the adequate cue(s) to offer was predefined and hand-scripted: it was connected to a database containing the different text resources and every text had a sequence of linguistic and semantic cues linked to it. Depending on the child’s condition, the agent’s automaton composes the dialogue utterances in order to include the appropriate support. We changed the utterances between the questions to avoid repetition in the agent’s dialogue: a replica is not executed if it has been delivered during the previous question.

Our current implementation had no natural language processing methods to condition the agent’s behavior: the recommendation system in the exploration space was only based on an automaton that shows the resources related to the topic of choice of the child. Also, the type of question entered by the child was only assessed later on during the data analysis phase.

_{Figure 2. System design}

Testing with children

The “Kids Ask” platform was tested with four classes from two different French elementry schools; participants were aged between 9 and 10. In order to investigate the efficiency of our approach (described above), we implemented two different versions of “Kids Ask” :

An experimental version : where participants interacted with “incentive” agents, i.e., agents that propose “knowledge gaps”
A control version : where participants interacted with “neutral” agents, i.e., agents that only ask if the child has a question about the text without giving any hints or cues.

Children’s ability to ask correct curiosity-based questions during the training: The results of the interaction showed that children that had the experimental version of “Kids Ask” (i.e., the incentive agents) asked significantly more divergent questions during their training than those who had the “neutral” agents. Questions were also of higher syntactic quality for participants who had the help of the incentive agents.

_{Figure 3. (A) Children's ability to ask curiosity-driven questions (i.e., higher-level divergent questions) is significantly better when having the support of the incentive agents (B) Children's ability to ask correct questions syntax-wise is significantly better when having the support of the incentive agents.}

Children’s ability to use curious question-asking to lead autonomous explorations of the pedagogical resources available: Our results showed that the incentive agents were more effective in helping children maintain autonomous explorations of the educational resources. We also showed that the exploration length was in line with children’s ability to generate the relevant curious questions. However, our results were not conclusive as to the patterns of explorations of these resources. Indeed, we saw similar entropies in the transitions between the resources that children from the two groups adopted during their information-searching cycles.
Children’s acheived domain-knwoledge learning progress: Finally, our results also showed a strong and positive relationship between the intensity of children’s curiosity-driven behaviors and their ability to acheive domain-knowledge learning progress.

Together, our results suggest that the more our agents are able to help children become aware of their knowledge gaps, the better they can maintain their curiosity-driven information-seeking behaviors and the more likely they are to acquire new knowledge independently.

Conclusion and future directions

With this project, we show that curiosity-driven behaviors such as curiosity-driven questioning and independent explorations can be trained on and enhanced with the help of educational technologies in general and conversational agents more particularily. We also show that these behaviors have a direct positive impact on the learning progress children can make. For future directions, we aim to work on the automation of our agents’ curiosity-prompting behaviors in order to facilitate their implementaion on larger scale and in different school activities. For this, we are studying the possibility to leverage the advances in the natural language processing field in order . Our work and results motivate the implementation of such approaches in both classrooms and online learning environments.

_{Cite this blog post}

@misc{,
  TITLE = ,
  AUTHOR = {Abdelghani, Rania and Oudeyer, Pierre-Yves and Law, Edith and de Vulipillieres, Catherine and Sauzeon, Helene},
  YEAR = {2022},
  HAL_ID = {hal-03810322},
  HAL_VERSION = {1},
}

_{Cite our IJCHS 2022 paper}

@article{ABDELGHANI2022102887,
title = {Conversational agents for fostering curiosity-driven learning in children},
journal = {International Journal of Human-Computer Studies},
volume = {167},
pages = {102887},
year = {2022},
issn = {1071-5819},
doi = {https://doi.org/10.1016/j.ijhcs.2022.102887},
url = {https://www.sciencedirect.com/science/article/pii/S1071581922001112},
author = {Rania Abdelghani and Pierre-Yves Oudeyer and Edith Law and Catherine {de Vulpillières} and Hélène Sauzéon},
keywords = {Human–computer interface, Cooperative/collaborative learning, Teaching/learning strategies, Improving classroom teaching}
}

_References

Abdelghani, R., Oudeyer, PY., Law, E., de Vulpillières, C., & Sauzéon, H. (2022). Conversational agents for fostering curiosity-driven learning in children. International Journal of Human-Computer Studies,Volume 167. [https://doi.org/10.1016/j.ijhcs.2022.102887]

Litman, J. (2005). Curiosity and the pleasures of learning: Wanting and liking new information. Cognition & Emotion, 19(6):793–814.

Von Stumm, HS., Chamorro-Premuzic., t. (2011). The hungry mind : Intellectual curiosity is the third pillar of academic performance. Perspectives on psychological science.

Metz, ke. (2008). Narrowing the gulf between the practices of science and the elementary school science classroom. The elementary school journal.

Jepma, M., Verdonschot, R., van Steenbergen, H., Rombouts, S., & Nieuwenhuis, S. (2012). Neural mechanisms underlying the induction and relief of perceptual curiosity. Frontiers in behavioral neuroscience.

Bulgren, JA., Marquis, JG., Lenz, BK., Deshler, DD., & Schumaker, JB. (2011). The effectiveness of a question-exploration routine for enhancing the content learning of secondary students. Journal of Educational Psychology, 103(3), 578–593. [https://doi.org/10.1037/a0023930]

Jirout, JJ., Vitiello, VE., & Zumbrunn, SK. (2018). Curiosity in schools. G. Gordon (Ed.), The new science of curiosity (pp. 243–265). Nova Science Publishers.

Ceha, J., Chhibber, N., Goh, J., McDonald, C., Oudeyer, PY., Kulic, D., & Law, E., (2019). Expression of Curiosity in Social Robots: Design, Perception, and Effects on Behaviour. CHI Conference on Human Factors in Computing Systems (CHI '19). Association for Computing Machinery, New York, NY, USA, Paper 406, 1–12. [https://doi.org/10.1145/3290605.3300636]

Gordon, G., Breazeal, C., & Engel, C. (2015). Can Children Catch Curiosity from a Social Robot? Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction (HRI '15). Association for Computing Machinery, New York, NY, USA, 91–98. [https://doi.org/10.1145/2696454.2696469].

Alaim, M., Law, E., Pantsado, KD., Oudeyer, PY., & Sauzéon, H (2020). Pedagogical Agents for Fostering Question-Asking Skills in Children. The 2020 CHI Conference on Human Factors in Computing Systems.

Humphries, J., & Ness, M. (2015). Beyond who, what, where, when, why, and how: Preparing students to generate questions in the age of common core standards. Journal of Research in Childhood Education 29, 551–561.[http://dx.doi.org/10.1080/02568543.2015.1073199]

Post, T. & Walma van der Molen, JH.,(2015). Development and validation of a questionnaire to measure primary school children’s images of and attitudes towards curiosity (the ciac questionnaire). Motivation and Emotion 42,1:159-178.[http://dx.doi.org/10.1007/s11031-018-9728-9]

Alven, V., Koedinger, KR., & Cross, K.,(1999). Tutoring answer explanation fosters learning with understanding. Artificial Intelligence in Education, IOS Press, 1999, pp. 199–206.

Roebers, C., von der Linden, N., & Howie, P.,(2007).Favorable and unfavorable conditions for children’s confidence judgments. British Journal of Developmental Psychology 25, 109–134. [https://doi.org/10.1348/980026151006X104392]

Gallagher, J. J., & Aschner, MJ.(1963). A preliminary report on analysis of classroom interaction. Merrill-Palmer Quarterly of Behavior and Development, 9(3), 183–194. [http://www.jstor.org/stable/23082786].

Loewenstein, G. (1994). The psychology of curiosity: A review and reinterpretation. Psychological Bulletin 116 75–98. [doi:10.1037/0033-2909.116.1.75].

Berlyne, DE. (1978). Curiosity and learning. Motivation and Emotion 2 91-175. [https://doi.org/10.1007/BF00993037].

Watching artificial intelligence through the lens of cognitive science methodologies

Tue, 19 Jul 2022 00:00:00 +0000

[Blog PDF] ● [ICLR 2022 paper] ● [Code]

This blog post presents our recent ICLR paper Language-biased image classification: evaluation based on semantic representations. The codebase accompanying this paper is available in the following repo.

Difficulty in interpreting functional mechanisms of a system

Humans tend to find human-like features in our daily life events. When winds blow a plastic bag on the street, children may feel like it is alive, known as animacy perception (Heider & Simmel, 1944; Scholl & Tremoulet, 2000; Tremoulet & Feldman, 2000). Artworks by Giuseppe Arcimboldo tell us that humans automatically detect human faces even if local components consist of objects unrelated to humans, like fruits or plants (Figure1A). Humans can also understand that other people have mental states like themselves, known as theory-of-mind (Baron-Cohen et al., 1985). These remarkable abilities enable us to read out someone’s subtle feelings and facilitate social communications in our daily life. However, they can get in the way when you interpret the behavior of an unknown system, e.g., when you are an AI researcher and would like to understand what representations are acquired in a machine learning model. You may unconsciously interpret a machine’s behavior as human-like from the similarity to daily human behaviors. But, it may be an overestimation due to our automatic prediction. We have to keep in mind that human interpretations tend to include a lot of expectations.

Recent machine learning models are trained using big data with massive parameters and achieve incredible performances for diverse tasks. Interpreting what representations are learned in these models is challenging due to their computational complexity. Even when you have a metric to evaluate system performance, quite different systems might output similar results for the metric. For example, in her ICLR 2022 keynote, Dr. Been Kim explained a clear example of the difficulty in interpreting machine learning models (the talk is available from this link) (Adebayo et al., 2018; Kim, 2022). The middle and right panels of Figure1B show saliency maps, which are metrics to explain which pixels machine learning models focus on when achieving an image classification task. Both saliency maps look similar and detect the main object of the photo, i.e., the bird. However, the right panel is actually the saliency map from an untrained network, while the middle one is from a trained network. This observation suggests that even when you see a visually meaningful output from a metric, you have to be cautious about interpreting the system.

_{Figure 1. (A) "Summer" by Giuseppe Arcimboldo (B) Saliency maps by trained and untrained networks (figure credit:
Been Kim, 2022, Beyond interpretability: developing a language to shape our relationships with AI(Kim, 2022),
link to blog)}

Considering the difficulty in interpreting systems, how can we understand the learned representations of machine learning models? We think the literature on perceptual and cognitive sciences in humans and animals offers clues to this problem. No magical methodologies exist in these research fields to answer whether a machine learning model acquires human-like intelligence. Even so, these fields are sensitive to interpretation bias and have focused on trying to avoid over-interpretation or mis-interpretation. We believe these methodologies in the perceptual and cognitive science litterature could also help us understand the mechanisms, properties and functions of machine learning models.

Our recent paper at ICLR 2022 (Lemesle et al., 2022) follows this approach, aiming to evaluate machine learning models through the lens of cognitive science methodologies. Specifically, we conduct cognitive science experiments for machine learning models (Lupker, 1979; Rosinski, 1977) and analyze the models’ internal representation using neuroscience methodologies(Kriegeskorte et al., 2008). By applying this approach to a recent machine learning model, CLIP (Radford et al., 2021), jointly trained with texts and images, we find that the vision and language do not appropriately share semantic representations in the model. This finding is unexpected, given that the model has been successful in many recent language-vision tasks (Bau et al., 2021; Patashnik et al., 2021). Researchers may achieve to develop machine learning models with remarkably higher performances for some distributions of tasks in the future, but they also have to consider how to interpret them deeply.

A key feature of recent advances in machine learning models is to utilize a large size of language models, and many researchers attempt to apply the models to various tasks. In this blog post, we first introduce how language processing contributes to acquiring various abilities in humans and machine learning models. Then, we explain a phenomenon, picture-word interference, observed both in humans and models. We argue how we evaluate the mechanisms underlying picture-word interference in models in our ICLR paper. By importing cognitive science methodologies, we evaluate models in a controlled way. We suggest that such a strategy prevents us from overestimating artificial intelligence due to partial and surface similarity between humans and models.

Language as a tool to shape perception and cognition

Humans use language not only as a tool to communicate with each other but also to shape fundamental aspects of perception and cognition. For example, color categorization emerges even before infants’ language acquisition (Yang et al., 2016), but language acquisition reorganizes these color categorical representations (Franklin et al., 2008). Humans also share common semantic representations for vision and language in some brain regions. For example, Quiroga et al. 2009 recorded single-cell activities from human patients, implanted with intracranial electrodes for clinical reasons. They show that single neurons in the medial temporal lobe respond selectively to representations of the same individual across the visual portrait and its written name.

Similar to advantages in humans, language has benefits in building autonomous agents capable of acquiring diverse skills in open-ended environments. For the design of Vygotskian autotelic AIs, a novel perspective suggested by Colas et al., 2022, agents are immersed into and interact with rich socio-cultural worlds, and then they internalize physical and sociocultural interactions within themselves (for the details, see also the website). They use language as a cognitive tool that mediates stimulus and actions to imagine their own goals, and for planning, reasoning, and learning about them. For example, in the IMAGINE approach (Colas et al., 2020), a Vygotskian autotelic AI approach, agents receive language descriptions from a social partner about their behaviors, internalize these descriptions, and create new goals they have never experienced by leveraging the compositionality of the internalized language information.

In addition, many joint learning models of language and vision have been reported recently. One prominent example is the CLIP (Contrastive Language–Image Pre-training) model reported by the OpenAI team (Radford et al., 2021), consisting of joint learning of language and vision, which we evaluated in our paper (Lemesle et al., 2022). It efficiently learns visual concepts from natural language supervision and can be applied to various visual tasks in a zero-shot manner.

_{Figure 2. Overview of the CLIP model. (figure credit: Alec Radford, et al., 2021, Learning Transferable Visual Models
From Natural Language Supervision (Radford et al., 2021),
link)}

The CLIP architecture consists of image and text encoders (Figure 2). These encoders are trained on a large dataset of image-text pairs with contrastive objectives, where they learn to align text and image representations for each pair (Figure 2, left). The pre-trained CLIP model can be used for various visual tasks. When applying the model for an image classification task (Figure 2, right), one uses the pre-trained text encoder to create a set of the embedding representations for classification labels by combining each label with a text prompt like “a photo of [label]”(Figure 2, (2)). For instance, for the labels “plane,” “car,” and “dog,” the text encoder outputs each embedding representation from the prompt “a photo of plane,” “a photo of car,” or “a photo of dog.” The pre-trained image encoder output the embedding representation for the input image (Figure 2, (3)). By computing the similarity of the image representation and each text label representation and selecting the highest similarity pair (e.g., the similarity between the image dog and text “a photo of dog”), the CLIP model can solve the image classification task in a zero-shot manner. Many applications of the CLIP model exist while combining the pre-trained model with other networks (Bau et al., 2021; Patashnik et al., 2021).

Picture word interference in humans and machines

While language contributes to acquiring general visual abilities, its abstraction can produce biased recognition for humans. Picture word interference is such a phenomenon. When humans observe an image coupled with a word having a different meaning from the image, the image categorization is disturbed by the word (Figure 3) (Lupker, 1979; Rosinski, 1977). For example, when a human observes a “dog” image with the written word “cat,” the written word “cat” interferes with the reaction time to answer the image category “dog” and increases the error rate.

_{Figure 3. Picture-word interference in humans and machines. Summary of Yoann Lemesle, et al., (2022); Languagebiased image classification: evaluation based on semantic representations (
Lemesle et al., 2022)}

In particular, the interference effect is strong when the image category is similar to the superimposed word. The finding suggests that shared semantic representations of images and words disrupt image categorization. It has been considered that multiple mechanisms mediate this effect in humans. First, when a participant observes a word-superimposed image, an activation process synthesizes semantic representations corresponding to the superimposed word and picture. This semantic representation is shared for pictures and superimposed words, so when the word is semantically similar to the picture, these activations are similar to each other. Based on the representations, a selection process decides which possible activation is the answer for the current task. Since the activated representation is shared across words and pictures, these dual activations confuse the decision in the selection process.

Similar to humans, recent joint models of language and vision also show recognition interference. For example, while the CLIP model can have high image classification performances for natural images in a zero shot manner (Figure 2), it can also be disrupted by superimposing a written word on the image, called a typographic attack (Figure 3)(Goh et al., 2021). In the case of Figure 3, the CLIP model recognizes the word-superimposed image as the “cat” image category.

Joint multimodal representations are bound to be very useful in many contexts. Even when you obtain noisy information from one modality (e.g., finding parking by car on a rainy day), information from another modality (e.g., the written word ’parking”) helps what you see. Picture-word interference is a special case of the functionality showing negative side effects. Therefore, showing similar interference to humans in machine learning models might look like a good sign that they also acquire the generic skill. However, it’s unclear what interference in machines means. If picture-word interference in machine models is due to a simple bias toward language information while ignoring visual information, the interference can result only in negative side effects.

Surface performance similarity does not always mean identical underlying mechanisms

Both humans and the CLIP model show picture-word interference. Observing a few similar biases in a picture-word interference task may lead to believing the mechanisms are the same. But, the surface performance similarity by partial observations may be due to largely different underlying mechanisms. The lens of cognitive science methodologies tells us that we should conduct a systematic, structured examination of the biases. Our study investigates this and shows that different functional mechanisms mediate picture-word interference in humans and the CLIP model (Figure 3).

_{Figure 4. Overview of our benchmark test.}

We first focused on the fact that human picture-word interference depends on the semantic relationship between images and written words. Cognitive science works show that the interference effect of written words on the image classification is larger when the image category is semantically similar to the word one (e.g., a written word “cat” for a “dog” image) (Rosinski, 1977). We imported this experimental paradigm for machine learning models and created the picture-word image dataset, in which we controlled the semantic categories between images and written words (Figure 4). Our dataset consists of a combination of natural image datasets and hierarchical superordinate/basic word labels. Our benchmark test is a 2 x 2 block design. One condition is the classification task type, indicating what is the image category level to be answered. The category level is superordinate (e.g., “animal,” “furniture”) or basic (“dog,” “cat”). The other condition is the superimposed word category level, superordinate or basic. In total, there are four conditions as follows.

The superordinate image classification for the superordinate word embedding (S/S).
The basic image classification for the superordinate word embedding (B/S).
The superordinate image classification for the basic word embedding (S/B).
The basic image classification for the basic word embedding (B/B).

We evaluated the CLIP model using the benchmark test. We showed that language-biased classification in the CLIP model does not depend on the semantic relationship between images and written words, although superimposing written words on images strongly biased the image classification to the written word category. The finding suggests that even when we observe picture-word interference in the CLIP model, the way to process the semantics of images and written words is different from humans.

To further explore what representations are acquired in the CLIP, we imported a neuroscience methodology tool, Representational Similarity Analysis (RSA) (Kriegeskorte et al., 2008), and investigated what is represented in the CLIP image encoder for word-superimposed images. RSA refers to an image-by-image similarity assessment of intermediate representations in a brain or model. By computing the representational similarity of images, one can understand which internal activations are similar. Through the analysis of images and written words, we found that the CLIP image encoder represents the neural representation of written words different from that of visual images (For example, the neural representation of a written dog is different from a visual dog image). Consistent results are also recently reported in another study (Materzynska et al., 2022).

Our study shows that the CLIP model does not have a common representation of language and vision, and the image classification is strongly biased toward written words (Figure 3). In this study, we imported cognitive science methodology to evaluate the models in a controlled manner. This approach prevents us from overestimating artificial intelligence due to partial performance similarity between humans and models.

Analogous to our picture-word interference example, previous works in comparative studies of humans and machines also see that interpretation from partial observations can lead to misunderstanding the underlying mechanisms of machine learning models and that we need to resist human interpretation bias (reviewed by Funke et al., 2021). For example, feedforward convolutional neural networks show high accuracies comparable to human performance in an image classification task. Although the classification accuracy is similar between humans and models, Geirhos et al. 2019 show that the models rely on texture features of an image, whereas humans rely on shape features, by conducting psychophysical experiments for humans and models.

In the future, people will expect more from artificial intelligence capabilities. The expectation can cause overestimations when interpreting the functional mechanisms of artificial intelligence. We may achieve to develop artificial intelligence showing remarkably higher performances for some distributions of tasks in the future. However, we cannot judge how generic they are without a strict way of interpreting them. We will need to further explore methodologies about how to interpret them.

_{Cite this blog post}

@misc{sawayama:hal-03729242,
  TITLE = ,
  AUTHOR = {Sawayama, Masataka and Lemesle, Yoann and Oudeyer, Pierre-Yves},
  URL = {https://developmentalsystems.org/watch_ai_through_cogsci},
  YEAR = {2022},
  MONTH = July,
  HAL_ID = {hal-03729242},
  HAL_VERSION = {v1},
}

_{Cite our ICLR 2022 paper}

@inproceedings{lemesle2021evaluating,
  title={Language-biased image classification: evaluation based on semantic representations},
  author={Lemesle, Yoann and Sawayama, Masataka and Valle-Perez, Guillermo and Adolphe, Maxime and Sauz{\'e}on, H{\'e}l{\`e}ne and Oudeyer, Pierre-Yves},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2022}
}

_References

Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. Advances in neural information processing systems, 31.

Baron-Cohen, S., Leslie, A. M., & Frith, U. (1985). Does the autistic child have a “theory of mind”? Cognition, 21(1), 37–46.

Bau, D., Andonian, A., Cui, A., Park, Y., Jahanian, A., Oliva, A., & Torralba, A. (2021). Paint by word. arXiv preprint arXiv:2103.10951.

Colas, C., Karch, T., Lair, N., Dussoux, J.-M., Moulin-Frier, C., Dominey, F. P., & Oudeyer, P.-Y. (2020). Language as a cognitive tool to imagine goals in curiosity driven exploration. NeurIPS 2020.

Colas, C., Karch, T., Moulin-Frier, C., & Oudeyer, P.-Y. (2022). Vygotskian autotelic artificial intelligence: Language and culture internalization for human-like ai. arXiv preprint arXiv:2206.01134.

Franklin, A., Drivonikou, G. V., Clifford, A., Kay, P., Regier, T., & Davies, I. R. (2008). Lateralization of categorical perception of color changes with color term acquisition. Proceedings of the National Academy of Sciences, 105(47), 18221–18225.

Funke, C. M., Borowski, J., Stosio, K., Brendel, W., Wallis, T. S. A., & Bethge, M. (2021). Five points to check when comparing visual perception in humans and machines. Journal of Vision, 21(3), 16–16. https://doi.org/10. 1167/jov.21.3.16

Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2019). Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations (ICLR).

Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021). Multimodal neurons in artificial neural networks [https://distill.pub/2021/multimodal-neurons]. Distill. https://doi.org/10. 23915/distill.00030

Heider, F., & Simmel, M. (1944). An experimental study of apparent behavior. The American journal of psychology, 57(2), 243–259.

Kim, B. (2022). Beyond interpretability: Developing a language to shape our relationships with ai. https://medium.com/ @beenkim/beyond-interpretability-4bf03bbd9394

Kriegeskorte, N., Mur, M., & Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2, 4.

Lemesle, Y., Sawayama, M., Valle-Perez, G., Adolphe, M., Sauzéon, H., & Oudeyer, P.-Y. (2022). Language-biased image classification: Evaluation based on semantic representations. International Conference on Learning Representations (ICLR).

Lupker, S. J. (1979). The semantic nature of response competition in the picture-word interference task. Memory & Cognition, 7(6), 485–495.

Materzynska, J., Torralba, A., & Bau, D. (2022). Disentangling visual and written concepts in clip. ´ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16410–16419.

Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., & Lischinski, D. (2021). Styleclip: Text-driven manipulation of stylegan imagery. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2085–2094.

Quiroga, R. Q., Kraskov, A., Koch, C., & Fried, I. (2009). Explicit encoding of multimodal percepts by single neurons in the human brain. Current Biology, 19(15), 1308–1313.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020. https://arxiv.org/abs/2103.00020

Rosinski, R. R. (1977). Picture-word interference is semantically based. Child Development, 643–647.

Scholl, B. J., & Tremoulet, P. D. (2000). Perceptual causality and animacy. Trends in cognitive sciences, 4(8), 299–309.

Tremoulet, P. D., & Feldman, J. (2000). Perception of animacy from the motion of a single object. Perception, 29(8), 943–951.

Yang, J., Kanazawa, S., Yamaguchi, M. K., & Kuriki, I. (2016). Cortical response to categorical color perception in infants investigated by near-infrared spectroscopy. Proceedings of the National Academy of Sciences, 113(9), 2370–2375.

Guiding and Learning Without Common Ground: a New Interactive Learning Paradigm

Wed, 27 Apr 2022 00:00:00 +0000

[Paper] ● [Code] ● [Video] ● [Website]

This blog post presents our recent ICLR paper Learning to Guide and to Be Guided in the Architect-Builder problem. The codebase accompanying this paper is available in the following repo. A recording of the paper presentation is also available on YouTube.

Humans are incredibly good at teaching and learning from each other. And impressively, they are able to do so in contexts where they share little to no common ground – such as a shared language or a known collective objective. In early stages of their development, children can interpret social cues from their parents to detect their intention and interact with them. In the following picture for instance, a preverbal infant that did not yet learn to produce words can learn to stack objects by reacting to guidance provided by a caregiver. In these types of interactions, not only does the child discover a new task, she also learns to interact and communicate with others.

_{A picture of a child learning to stack cubes from interactions with a caregiver (image taken from here)}

It turns out that social interactions are central to child development [1]. Long before they learn about language, children constantly adapt to the social context in which they evolve. For instance, they demonstrate ‘social sensitivity’ adapting their behavior to emotions [2] (see this video). They also detect pedagogical signals and are able to infer the communicative intention of a caregiver [3] (see this recent paper [4] section I.A for a thorough list of children learning properties)

Therefore it seems that pre-verbal communication enables children to acquire new skills in absence of direct reinforcement signals, demonstrations or even a shared communication protocol. In the quest to design interactive artificial agents, such flexible learning skills look like a desirable feature. But how do we make progress in this direction? How can artificial agents figure out what to do and learn relevant behaviors without any common ground? On the one hand, HRI focuses on human teaching robots and investigates how a robot-learner can figure out what a human-teacher wants it to do [5] [6]. On the other hand, Multi-Agent Reinforcement Learning (MARL) [7] can model robot-robot interactions by placing multiple agents in an environment and training them to solve collaborative tasks. Yet, in MARL, all agents have access to a reward signal defined by the collaborative task they need to achieve. This assumption is not really representative of a child interacting with a caregiver since the reward is an external signal shared between agents. The question is then how can we come up with a realistic setting where two agents that share no common ground can learn a communication protocol to solve a task?

We propose to answer this question by drawing inspiration from Experimental Semiotics [8]. Experimental Semiotics is a discipline that aims at investigating, in the laboratory, how humans develop common grounds in order to collaborate. It uses experiments to model the development and the emergence of communication protocols. However, interactions between humans are complex and involve many dimensions (symbols, eye gaze, gesture, posture). To get a fine grasp of the underlying mechanisms at hand, experimenters try to study these dimensions in isolation.

In this blog post, we will review a recent experiment studying the emergence of communication between humans in a co-construction task involving lego blocks. We will then present the Architect-Builder Problem (ABP), a framework that draws inspiration from this experiment to propose a new interactive learning paradigm for artificial agents. We will then discuss potential real world applications of the ABP before moving on to the study of Architect-Builder Iterated Guiding (ABIG), a solution to the ABP.

The Co-Construction Game

The Co-Construction game is an experiment proposed by Vollmer et al. [9]. It aims at studying how humans can align and create common ground to solve a collaborative task. The game involves two players: an Architect and a Builder, each having a separate role. Their goal is to build a lego construction. Agents are located in separate rooms. The Architect has a picture of the construction target but cannot manipulate the blocks. In the other room, the builder can move the blocks around but does not know what it needs to build. The two agents can interact via a constrained communication channel. The architect can press buttons in order to display abstract symbols on the screen of the builder’s room. The architect can also see the builders’ workspace thanks to a video stream.

_{The Co-Construction game specifications: link to website and link to video.}

Communication is made challenging because agents only interact with abstract symbols that have no a priori meanings. In order to succeed, a pair must progressively establish a communication convention – or protocol – through the interplay and history of their mutual interactions. The builder needs to simultaneously learn the task and the meaning of messages while the architect has to adapt to the builder’s reactions.

This game is often deemed impossible or extremely hard by the participants. What about you? Would you expect to succeed? It seems pretty hard to associate meanings to messages on the fly while performing a task. To demonstrate this concept, Jonathan Grizou developed this online game where you need to crack the code of a vault by learning the meaning of the interface buttons. In the Co-Construction game, it turns out that the subjects did succeed but it took time for them to agree on a protocol (approximately 20 minutes). The solution they agree upon is not so advanced. Successful dyads always converge to a positive/negative type of protocol similar to the Hot and Cold game we used to play as children. An example of an interaction in a face to face scenario is given in the following video (note that the participants are instructed to remain impassive and only communicate with the provided tools).

_{Interaction in a face to face variation of the Co-Construction Game}

In a recent ICLR paper [10], we investigate if such behaviors can be observed between artificial agents. For this purpose we translated the Co-Construction game into a sequential decision making formalism that we coined the Architect-Builder Problem.

The Architect-Builder Problem

To stick with the spirit of the Co-Construction game, we designed a simple construction environment called BuildWorld. BuildWorld is a 2D grid world in which the builder can navigate and manipulate blocks to construct various shapes. In the ABP, the two agents have asymmetrical roles. Only the architect knows about the target construction (the goal). In a Markov Decision Process, this means that only the architect perceives the rewards given by the environment. However, the architect cannot make any actions in the environment. It can only observe the states ($s$) of the environment. The interactions unfold in the following sequence. First, the architect produces a message ($m$) given a certain state and a goal. The builder then receives the message and takes an action ($a$) that makes the environment transition to a new state ($s’$).

_{Sketch of the Architect-Builder Problem and sequential diagram of interactions.}

In this new multi-agent interactive learning setting where only one agent perceives the reward, we make the assumption that the Architect has access to the environmental reward $r$ and transition functions $P_E$. In less formal terms, this means that the Architect knows about the dynamics that govern the manipulations of blocks. We believe that this assumption is sound and matches with the conditions in which the Co-Construction game is performed where human architects know how to stack blocks in desired configuration even if they cannot physically do it themselves.

At this point of the post you might say: OK the ABP is a faithful implementation of the Co-Construction game but why does it make it an interesting learning setup for artificial agents? There are several reasons for that.

First it is interesting because it is a new way to approach multi-agent learning. Most of recent research [7] study multiple agents that have the same learning mechanism to learn to solve a collaborative task. Crucially, in such a setup, all the agents perceive the reward associated with the task and act to influence the environment transitions. In our case, we consider agents with different learning mechanisms where one totally ignores the task to perform while the other is aware of the task but cannot directly drive the environment towards its completion. We believe that these constraints make the ABP a challenging and realistic setup to study the emergence of communication.

→

_{Comparison between classical MARL (left) setup and the ABP (right).}

We like to think that the ABP is not only designed to study the emergence of communication for collaboration. It is also a relevant test bed to foster fundamental research around the mechanisms underlying interactive teaching and interactive learning. In fact, the recent IGLU NeurIPS Competition proposed to investigate these questions in a setup similar to ours with an Architect and a Builder interacting with natural language in a 3D-rendered construction environment.

ABP is interesting because it has the potential to model real world applications. Indeed, it shares similar assumptions with Brain Computer Interfaces (BCI). A typical example of a BCI is a robotic arm controlled by brain signals.

_{An example of a BCI (image taken from here).}

With such systems, it is impossible to anticipate the intentions of users and the tasks that they will try to perform. Thus, we cannot devise a reward signal that would help the computer and the brain agree on a communication code. A solution to this problem is to calibrate the machine. This is often done by using supervised learning to map brain signals to a computer interpretable code that is later used during interaction. However, this calibration phase is time consuming and depends on the user and the task. On top of that, the initial placement of sensors that read user signals may move over time requiring re-calibration. Recent research proposes to tackle this problem via Interaction-Grounded Learning [11], where “the learner must deduce a grounding for the feedback solely via interaction” just as the builder needs to learn to solve the task solely through the messages it receives by the architect in the ABP. This can enable online-calibration of BCIs [12] where (1) only the human (the brain) knows what she wants to do ; (2) only the robotic arm can act in the environment and (3) the mapping from brain signals to arms movement is not know a apriori but must emerge from the interactions between the human and the BCI.

Finally, the Architect-Builder Problem should be differentiated from Hierarchical Reinforcement Learning (HRL). In ABP the builder does not have access to any reward while in HRL the builder receives a reward given by the architect. So HRL assumes that there is a “reward channel” through which the architect can explicitly set the learning signal of the builder. In the case of a Brain-Computer Interface (BCI) where a user must control a prosthetic arm using only brain signals, HRL would require us to assume that the user is directly communicating a reward signal to the robotic arm. In other it would rely on a way to translate brain signals into scalar values indicating how much the user is satisfied with what the arm is doing. This is precisely the non-trivial calibration problem that BCIs are trying to address. Another way of looking at it is to note that the HRL setting requires to pre-define the communication protocol between the agents (i.e. “messages are scalars that must be interpreted as a reward and maximized by the builder”) and does not have agents negotiate and invent one.

Architect-Builder Iterated Guiding (ABIG)

We have seen that learning a task from interaction without reward is an interesting problem in different scientific fields including experimental semiotics and AI, as well as applications such as BCI. Let’s now go over our proposed solution to efficiently solve this problem.

The ABP contains two main challenges:

Non stationarity: Because both the architect and the builder are learning at the same time. Their respective learning environments include the learning dynamics of each other.
No reward: The Builder never perceives a reward. It has no learning signal so how can the Builder learn?!

To address these two challenges we rely on two priors:

To recover stationarity, we break down learning into Interaction Frames (or Pragmatic Frames) [13] during which each agent learns sequentially in turn. Pragmatic Frames provide a learning environment for children during which information is structured in recurrent and sequential interactions. It has been shown that it facilitates the emergence of language [14].
To find a learning signal for the Builder we use the Shared Intent prior. We assume that both the Architect and the Builder aim at maximizing the same objective (solving the task at hand).

Using these two priors we introduce the Architect-Builder Iterated Guiding algorithm. ABIG is an iterative method that alternates between a Modeling and a Guiding frame.

_{Diagram of the ABIG algorithm.}

During the first frame (the modeling frame) the Builder is fixed and the Architect sends random messages to observe the Builder’s reaction. Doing so, it constructs a dataset $(s,m,a,s’)_t$ and learns a model of the builder $\tilde{\pi}_B$ using Behavioral Cloning (BC)

In the second frame (the guiding one) the architect uses the model of the builder alongside with the reward signal to plan and pick the messages that will maximize the reward. In other words, it will select messages that would yield actions by the builder that would generate high rewards. Once the architect has produced a message, the builder receives it and picks an action according to its policy $\pi_B$. During these interactions the builder stores the guiding data $(s,m,a,s’)_t$ and this is when the shared intent prior comes into play. If the builder assumes that the guiding trajectory generated by the architect maximizes the reward, this means that the guiding trajectory that it just stored are in fact demonstrations of how to solve the task. Therefore the builder can use Self-Imitation (doing BC on the guiding trajectory) to improve its policy.

By iterating between these two frames, we show in our paper that Architect-Builder pairs can learn successful communication protocols that not only allow them to solve construction tasks but that can also solve new tasks in zero-shot transfer. Below are rollouts of interactions leveraging a protocol (learned by placing a block at a given location) to construct multiblock structures with shapes “Y” and “O”. Interestingly, throughout learning, agents converge to very simple protocols where the builder’s actions are led by messages rather than states (the mutual information between states and actions decreases while the one between messages and actions increases), meaning that the architect is suggesting actions to the builder.

_{Example of interactions using successful protocols learned with ABIG.}

Conclusion

In this work we propose a versatile new learning paradigm well suited to study for example the constrained emergence of communication or the calibration of Brain-Computer Interfaces. It is exciting to realize all the possible avenues towards which this setting could be extended: more complex tasks, varied communication channel and recurrent policies for more complex communication protocols; relaxing the sequentiality between modeling and guiding frames to allow for online and continual learning, having the builder internalize the solved task so that it can be autonomous and not rely on architect’s messages anymore, etc.

More fundamentally, ABP is a first try toward shifting the learning paradigm from specified conventions/interfaces and rewards as learning signals to more flexible and negotiable social supervisions. This resembles how we humans co-adapt to using many gestures in a personalized way when working with each other.

_{Cite this work}

@inproceedings{
  barde2022learning,
  title={Learning to Guide and to be Guided in the Architect-Builder Problem},
  author={Paul Barde and Tristan Karch and Derek Nowrouzezahrai and Cl{\'e}ment Moulin-Frier and Christopher Pal and Pierre-Yves Oudeyer},
  booktitle={International Conference on Learning Representations},
  year={2022},
  url={https://openreview.net/forum?id=swiyAeGzFhQ}
}

_References

L. S. Vygotsky. Thought and Language 1934.
T. M. Field, R. Woodson, D. Cohen, R. Greenberg, R. Garcia, K. Collins. Discrimination and Imitation of Facial Expressions by Term and Preterm. Infant Behavior and Development, 6(4), 485-489. https://doi.org/10.1016/S0163-6383(83)90316-8.
G. Csibra, G. Gergely. Natural Pedagogy. Trends Cogn Sci. 2009 Apr;13(4):148-53. doi: 10.1016/j.tics.2009.01.005. Epub 2009 Mar 13. PMID: 19285912.
O. Sigaud, A. Akakzia, H. Caselles-Dupré, C. Colas, P.Y. Oudeyer, M. Chetouani. Towards Teachable Autotelic Agents. 2021.
T. Cederborg. Artificial Learners Adopting Normative Conventions From Human Teachers. Journal of Behavioral Robotics 2017.
J. Grizou, M. Lopes, P.-Y. Oudeyer. Robot learning simultaneously a task and how to interpret human. ICDL 2013.
R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, I. Mordatch. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments. NeurIPS 2017.
B. Galantucci and S. Garrod. Experiment Semiotics: a Review. Frontiers in Human Neuroscience 2011.
A.-L. Vollmer, J. Grizou, M. Lopes, K. Rohlfing, P.-Y. Oudeyer Studying the Co-Construction of Interaction Protocols in Collaborative Tasks with Humans. ICDL 2014.
P. Barde, T. Karch, D. Nowrouzezahrai, C. Moulin-Frier, C. Pal, P.-Y. Oudeyer. Learning to Guide and to Be Guided in the Architect-Builder Problem. ICLR 2022.
T. Xie, J. Langford, P. Mineiro, I. Mommennejad. Interaction-Grounded Learning. ICML 2021.
J. Grizou, I. Iturrate, L. Montesano, P.-Y. Oudeyer, M. Lopes. Calibration-Free BCI Based Control. Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, AAAI'14.
A.-L. Vollmer, B. Wrede, K. J Rohlfing, P.-Y. Oudeyer. Pragmatic frames for teaching and learning in human–robot interaction: Review and challenges. Frontiers in neurorobotics, 10:10, 2016.
J. Bruner. Child’s talk: Learning to use language. Child Language Teaching and Therapy 1(1):111–114, 1985.

Developmental Systems, a Blog of the Flowers Lab

When LLMs discover the world

1) What is a world model?

2) Building world models

3) WorldLLM: curious LLMs that explore and refine their world model

4) From prediction to understanding

Acknowledgements

References

Learning together: how others can fuel (or hinder) our curiosity

A story of “motivating” cakes

Intrinsic and extrinsic motivation: an inseparable duo

Self-Determination Theory: when the social context comes into play

Learning by observing others: social learning

Learning under group pressure: conformity

Learning through the mere presence of others

The Flowers team in this landscape: the first milestones

Conclusion

References

The Pleasure of Getting Better: Is Learning a Basic Need?

When LLMs Play the Telephone Game: An Interactive Tool

The Phylogenetics of Artifacts - A deep dive into the evolution of cultural objects, artificial life forms and language models

Part I: What is Phylogenetics ?

Methods to reconstruct phylogenetic trees in biological systems

How to select the genes ?

Gathering genetic information

Population comparison

Phylogenetic tree

Part 2: Theory of Evolution

DNA: the functional and archeological support for evolution

What makes a good marker of evolutionary history?

The limits of tree-based models

Part 3: Phylogenetics Beyond Biology

Cultural evolution

Cultural artifacts and the extended phenotype (Dawkins)

Myths and oral traditions

What is the “DNA” of an artifact?

Part 4: LLM Phylogenetics

What is a LLM?

LLMs as populations of text

Genes and alleles in LLMs

Selective pressure and drift in LLMs

PhyloLM: results and interpretation

Limitations and open questions

Can AIs understand our world? Functionally grounding LLMs in interactive environments.

The symbol grounding problem

Towards functional grounding

BabyAI-Text, an experimental testbed for functional grounding

GLAM, functionally grounding through online interactions

Towards grounded LLMs in robots?

An ecological approach to Artificial Intelligence

Subscribe to our Twitter.

Designing artificial conversational agents to train children's curiosity during learning, a proof of concept through the Kids Ask project

Curiosity : a key factor for enhancing learning experiences and outcomes

Curiosity during learning : the major brakes

The “Kids Ask” project

Design

Technical implementation

Testing with children

Conclusion and future directions

Watching artificial intelligence through the lens of cognitive science methodologies

Difficulty in interpreting functional mechanisms of a system

Language as a tool to shape perception and cognition

Picture word interference in humans and machines

Surface performance similarity does not always mean identical underlying mechanisms

Guiding and Learning Without Common Ground: a New Interactive Learning Paradigm

The Co-Construction Game

The Architect-Builder Problem

Architect-Builder Iterated Guiding (ABIG)

Conclusion