Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models

Logan Cross, Violet Xiang, Agam Bhatia, Daniel Yamins, Nick Haber,
Stanford University,
Paper arXiv Code

Hypothetical Minds is a LLM agent for multi-agent settings that generates hypotheses about other agents’ strategies in natural language, enabling it to outperform RL and LLM agent baselines across collaborative, competitive, and mixed-motive domains

Hypothetical Minds integrates partially-observable inputs in text, a memory module, and LLM-based Theory of Mind and Subgoal modules to generate high-level strategies and action plans to achieve the strategies. The Theory of Mind module generates hypotheses about other agents' strategies in natural language, and evaluates and refines these hypotheses by reinforcing hypotheses that make accurate behavioral predictions. High-level strategies are then generated conditioned on the best hypothesis. The Subgoal module breaks down high-level strategies into concrete subgoals using action functions, given the current state.


Abstract

Multi-agent reinforcement learning (MARL) methods struggle with the non-stationarity of multi-agent systems and fail to adaptively learn online when tested with novel agents. Here, we leverage large language models (LLMs) to create an autonomous agent that can handle these challenges. Our agent, Hypothetical Minds, consists of a cognitively-inspired architecture, featuring modular components for perception, memory, and hierarchical planning over two levels of abstraction. We introduce the Theory of Mind module that scaffolds the high-level planning process by generating hypotheses about other agents' strategies in natural language. It then evaluates and iteratively refines these hypotheses by reinforcing hypotheses that make correct predictions about the other agents' behavior. Hypothetical Minds significantly improves performance over previous LLM-agent and RL baselines on a range of competitive, mixed motive, and collaborative domains in the Melting Pot benchmark, including both dyadic and population-based environments. Additionally, comparisons against LLM-agent baselines and ablations reveal the importance of hypothesis evaluation and refinement for succeeding on complex scenarios.

 

 

 

Theory of Mind (ToM) Module for Running With Scissors. This cognitive module receives input in the form of interaction history and outputs a target inventory as a goal for the Subgoal module. Information is processed in 5 steps, including using the available information to generate, evaluate, and refine hypotheses about the opponent’s strategy.

Results