AgentWorld: Benchmarking Long-Horizon Collaboration of Multi-agent LLMs

TL;DR

The paper introduces AgentWorld, a benchmark for evaluating long-horizon collaboration of multi-agent LLMs. It proposes a graph-based metric, Causal Collaboration Effectiveness, to quantify collaboration effectiveness. The benchmark reveals systematic failure modes in current models.

Why It Matters

Evaluating long-horizon collaboration is crucial for real-world applications, such as team management and decision-making. The proposed benchmark and metric can help improve the performance of multi-agent systems. This can lead to more effective human-AI collaboration and decision-making.

Key Findings

• communication breakdowns
• role confusion
• inability to maintain shared plans