History Problem Framework Missions Metrics Findings Impact Deep Dive Quiz
Interactive Paper Explainer

Agents That Cooperate —
and Compete

A visual, step-by-step guide to MultiAgentBench and its MARBLE framework — the benchmark that scores LLM agent teams on what they achieve together AND on how well they coordinate, communicate, and compete along the way.

Start Learning Read the Paper ↗
6
Scenarios (4 Co-op + 2 Competitive)
100
Test Cases per Task
5
Metrics Tracked
2025
Year Published
History

From Frameworks to a Benchmark

Multi-agent LLM systems exploded in 2023–2024 — but almost nobody was measuring them. Here is the road that led to the first benchmark built for both collaboration and competition.

2023 · Mar
CAMEL (Li et al.)
Role-playing "communicative agents" explore task cooperation — two LLMs chatting until a job is done. Evaluation stays ad hoc.
2023 · Aug
MetaGPT (Hong et al.)
Encodes human standard operating procedures into a role pipeline — product manager → architect → engineer. Impressive builds, no standard scores.
2024
Magentic-One & MAS surveys
Microsoft's generalist multi-agent orchestrator arrives as surveys map the field — evaluation of coordination itself remains fragmented.
2024
Single-agent benchmarks mature
Benchmarks like AgentBench grade one LLM against one environment — team dynamics, competition, and credit assignment are out of scope.
2025 · Mar
🚀 MultiAgentBench (Zhu et al.)
The comprehensive benchmark for LLM multi-agent systems: six interactive scenarios, mixed motives, milestone-based KPIs, plus the MARBLE framework.
2025 →
Mixed-motive MAS research
Game-theoretic dynamics, emergent social behavior, and credit assignment become measurable study objects — not anecdotes.
Key Insight

Multi-agent papers measured whether the team finished — never how. MultiAgentBench turns coordination itself into data: which milestones were hit, by whom, at which turn, and how well the agents talked and planned along the way.

WHAT CHANGED IN EVALUATION
old: final_answer → score
new: turn → action → milestone? → who? → KPI + C/P/CS
Task outcome AND coordination quality, per turn, per agent.
Chapter 01

The Problem: Scores Without Coordination

Before MultiAgentBench, "multi-agent evaluation" usually meant grading the final artifact of a team that was only ever supposed to cooperate. Two blind spots, one benchmark.

🧾
Team-Score-Only Evaluation
  • Benchmarks grade the final output — or a single agent in isolation
  • Scenarios are cooperative-only; competition is ignored
  • No per-agent credit: who did the work stays invisible
  • A team that fails slowly looks identical to one that never started
  • Coordination quality is never scored — just assumed
🧭
MultiAgentBench's Solution
  • Turn-level milestone detection logs hits AND contributing agents
  • Mixed motives: mutual-goal AND conflicting-goal scenarios
  • Milestone KPI measures each agent's share of the work
  • Communication and planning each get a 5-point LLM-judged score
  • Task scores via rule-based checkers or LLM rubrics — per scenario
Analogy — Sports Analytics

A box score that only records the final result tells you who won, not how. Real analytics added passes, assists, and turnovers — and suddenly you could tell a coordinated win from a lucky one. MultiAgentBench is the stat sheet for agent teams: goals AND passes, milestones AND coordination.

old scoreboard:  final_score = 1 …0  (was it teamwork or luck?)
new scoreboard:  milestones 4/4 · assists A=2 B=2 · passes 12 · turnovers 1
Chapter 02

MARBLE — a Coordination Backbone

The benchmark runs on MARBLE: Multi-agent cooRdination Backbone with LLM Engine. Planners strategize, actors execute, and every turn flows through shared state.

The Name, Decoded
Multi-agent cooRdination Backbone with LLM Engine

MARBLE is both the benchmark harness and a reusable framework: it initializes agents with personas and roles, wires them into a communication graph, runs the environment turn by turn, and logs everything the metrics need.

Coordination Engine
Orchestrates execution: initializes agents, tasks, and relationships via a Configuration Module, then keeps every module in sync.
Agent Graph
A structured graph of who may talk to whom — agents can only communicate through explicitly defined relationships.
Cognitive Module
Maintains each agent's persona, inter-agent relationships, and reasoning strategy as the session evolves.
Environment Module
The simulated world: agents act through a function-calling interface, choosing an action at ∈ F at each time step t.
Communication
Structured dialogue between agents — negotiating roles, coordinating plans, balancing collaboration with competition.
Action + Memory
Executes plans via function calls; outcomes feed straight back into individual and shared memory stores.
Two Kinds of Agent
  • Planners devise task inputs, strategize, and manage task allocation
  • Actors execute: they interact with the environment and other agents through available tools
  • How planners and actors are wired together defines the coordination protocol
per turn t: agent → action at ∈ F
→ environment updates shared state
→ observation → memory → next agent
(max iterations: 5 in research, 20 in Minecraft)
Four Coordination Protocols
star
One central planner assigns tasks to all actors — strong oversight, limited scalability.
tree
Hierarchical: a top planner delegates to sub-planners, which coordinate actors.
graph-mesh
Interconnected actors communicate directly — concurrent, distributed decisions.
chain
Sequential handoff: each agent passes its decision to the next — good for dependent steps.
Interactive Demo — Mixed-Motive Mission: The Last Egg

Two agents, one kitchen, one last egg. Advance the mission turn by turn and watch what a shared-resource world does to "cooperation" — and how MARBLE-style logging turns it into milestones, conflicts, and per-agent credit.

Illustrative simulation of MARBLE's turn-based loop and shared-resource bargaining — not real paper transcripts.

Chapter 03

Six Missions, Two Motives

Four task-oriented scenarios share a mutual goal. Two social-simulation scenarios set agents against each other. Every task ships with 100 test cases, built from adapted prior work plus LLM-generated tasks verified by humans.

🔬 Research Co-Authoring
mutual goal LLM rubric
Agents with complementary research profiles co-author a proposal in the structured 5-question (5q) format, using tools like get_related_papers.
⛏️ Minecraft Building
mutual goal rule-based
Agents collaboratively construct structures in a shared world — scored by block hit rate, mapped to a 5-point scale.
🗄️ Database Error Analysis
mutual goal rule-based
Exactly five agents, each specializing in a distinct root cause, debug a live PostgreSQL running in Docker — talking over a graph structure while querying system views.
💻 Coding Challenges
mutual goal rule-based
Agents with complementary skills — debugging, execution, test-writing — develop a complete, modular software solution.
🐺 Werewolf
conflicting goals rule-based
The social-deduction game: hidden roles, deception, and voting. Score = surviving villagers − surviving werewolves. Roles include seer, witch, guard, and sheriff.
🤝 Bargaining
conflicting goals LLM rubric
Negotiations over shared resources: agents strive to maximize individual gains through strategic concessions or alliances.
The Scenario Suite at a Glance
ScenarioMotiveTeam SetupTask Score
Research🤝 CollaborateComplementary research profiles (1–7 agents tested)LLM scoring rubric
Minecraft🤝 CollaborateShared-world buildersBlock hit rate
Database🤝 CollaborateExactly 5 root-cause specialistsRule-based accuracy
Coding🤝 CollaborateComplementary coding skillsRule-based
Werewolf⚔️ CompeteVillagers vs werewolves, hidden rolesSurvivor margin
Bargaining⚔️ CompeteNegotiators over shared resourcesLLM scoring rubric

Scenarios mix two provenance routes: tasks adapted from prior work or datasets (database, research) and LLM-generated tasks refined with human verification (Werewolf, Bargaining).

Interactive Demo — Milestone Anatomy

Missions are decomposed into flexible milestones — and each milestone is credited to the agents who contributed. Click through the research mission's milestone graph to see the sub-goals, the contributors, and the checker rule.

Simplified from the paper's research scenario. In MARBLE, an LLM-based detector monitors every iteration and logs which milestones were achieved and by whom — that log becomes the KPI.

Chapter 04

Scoring How, Not Just What

Two score families: task completion (milestone KPI + task score) and coordination (communication + planning). Together they grade the mission AND the teamwork.

KPIoverall = (1/N) Σj KPIj = (1/(N·M)) Σj nj
N
Agents
Number of agents in the session — e.g., exactly 5 in database analysis; 1, 3, 5, 7 tested in research.
M
Milestones
Total milestones the task is segmented into — flexible, e.g., "five key queries (5q) drafted" or a previous 5q enhanced.
n_j
Contributions
How many milestones agent j actually contributed to — logged turn by turn by the LLM detector.
KPI_j
Share of Work
n_j / M — one agent's slice of the mission. The overall KPI averages it across the team.
🎯 Milestone KPI
Credit assignment from the milestone log: who hit what, and when. Averages each agent's contribution ratio.
📝 Task Score (TS)
Final output quality — LLM rubric for research and bargaining; rule-based checkers for Minecraft, database, coding, Werewolf.
💬 Communication (C)
LLM-judged on a 5-point scale from profiles + aggregated chat. If no communication occurs, C = 0 by rule.
🗺️ Planning (P)
LLM-judged on a 5-point scale: organizing tasks, maintaining roles, adapting strategy from plan logs.
🤝 Coordination (CS)
The headline collaboration number: the average of C and P. The paper also ran human evaluation to check alignment with these metrics.
Why Milestones Instead of Checkpoints

Milestones are deliberately broadly defined, not rigid checkpoints: "completing five key queries (5q) for the research proposal" counts — and so does enhancing an earlier 5q set. A language model watches each MARBLE iteration, decides whether m1, m2, … have been achieved, and logs the contributing agents — combining human- or LLM-generated outlines with dynamic, execution-based assessment.

Chapter 05

What the Experiments Found

Five models — Llama-3.1-8B, Llama-3.1-70B, Llama-3.3-70B, gpt-3.5-turbo, gpt-4o-mini — ran the full benchmark. The surprises were not about raw task scores.

GPT-4O-MINI
84.13
Research TS — highest single-scenario score; best average TS overall
Not the largest model — model ability still drives task completion
MINECRAFT TS CEILING
33.60
best Task Score ANY model achieved — collaborative building is brutally hard
5-point block-hit-rate scale (×20)
WEREWOLF TS CEILING
36.33
best TS any model — by Meta-Llama-3.3-70B; deception stays hard
Social deduction punishes almost every team
COGNITIVE PLANNING
+3%
milestone achievement rate from cognitive self-evolving planning
Plus the best Coordination Score of any strategy
Average Task Score (TS) per Scenario — Real Numbers, Table 1 of the Paper
ModelResearchMinecraftDatabaseCodingBargainingWerewolf
Meta-Llama-3.1-8B80.876.1234.0059.9072.8112.64
Meta-Llama-3.1-70B80.800.2153.0062.1072.1319.82
Meta-Llama-3.3-70B80.009.1528.5056.6073.1536.33
gpt-3.5-turbo70.205.0545.0055.5071.6715.69
gpt-4o-mini84.1333.6045.0065.1074.4714.06

For Minecraft, Database, Coding, Bargaining, and Werewolf the paper reports its 5-point scales multiplied by 20; research TS comes from an LLM rubric. gpt-4o-mini takes the highest average TS — and in three scenario pairs the same model tops TS and CS, making CS a good indicator of task success.

🕸️ Graph topology wins
In the research scenario, the graph protocol earned the best task performance, planning efficiency, and token usage; star matched it on task score. Tree was worst — highest tokens, lowest TS and CS.
👥 The group-discussion paradox
Counterintuitive: group discussion scored the worst across all metrics — an overly large planning group hinders effectiveness, like oversized real-world organizations.
🧠 Self-evolving planners
Cognitive evolving planning (Reflexion-style feedback) delivered the best CS with a task score comparable to CoT — and +3% milestone achievement.
👥 More agents ≠ more KPI
Scaling the research team 1 → 3 → 5 → 7 agents lowered the overall KPI: coordination complexity grows faster than the work. CS jumped most from 1 to 3 agents.
⏱️ Iteration sweet spot
In Minecraft, task and coordination scores climbed from 1 to 7 max iterations, then dropped sharply at 10 — coordination flatlined beyond 7.
🎭 Emergent "aha-moments"
Roles evolved mid-game: a Seer turned leader, a Witch moved from hoarding to risk-taking — and one Seer identified the werewolf on night one but never said it publicly, dooming the villagers.
Interactive Demo — Coordination Strategy Lab

Pick a planning strategy for the team, run the mission, and compare milestone rate, communication, and planning scores. The ordering matches the paper's findings — the exact bar heights are illustrative.

Legacy

Impact — Mixed-Motive Evaluation

MultiAgentBench made coordination measurable and competition benchmarkable. Code and datasets are public — the field got shared infrastructure, not just a leaderboard.

🎭 Mixed-motive standard
One benchmark covering collaboration AND competition — cooperative teamwork and adversarial self-interest scored on the same scale.
🧭 Milestone-based KPIs
Per-agent credit assignment: which milestones were hit, by whom, at which turn. Coordination failures became visible, not hypothetical.
🏗️ MARBLE infrastructure
An open-source coordination backbone — 4 topologies, 4 planning strategies, 6 environments — reusable well beyond the paper's experiments.
💡 Strategy insights
The group-discussion paradox and cognitive-planning gains give practitioners evidence for how to structure agent teams, not just whether to build them.
♟️ Game theory meets LLMs
Bargaining and Werewolf turn negotiation, deception, and alliance-formation into reproducible evaluations — a bridge from MAS research to LLM agents.
🔭 Emergent-behavior observatory
Turn-level logs let researchers study "aha-moments": role shifts, trust polarization, and false consensus — with data instead of anecdotes.
What It Did NOT Solve
Deep Dive

Grade the Journey, Not the Destination

Binary task success tells you who finished; milestone-based scoring tells you how. MultiAgentBench's KPI framework assigns credit at intermediate checkpoints — so a team that got 80% of the way and a team that failed instantly stop looking identical. On hard scenarios (Minecraft 33.60, Werewolf 36.33), the milestone view is the only informative one.

📏
What Milestones Buy
  • Six scenarios: research, Minecraft, database, coding (mutual goal) + Werewolf, bargaining (conflicting goal) — 100 cases each
  • MARBLE backbone: planners vs actors, 4 topologies, 4 planning strategies — a controllable experiment grid
  • Milestone KPI for credit assignment + TS, plus 5-point Communication / Planning / Coordination scores
  • Turn-level logs expose emergent behavior — Seers turning leaders, Witches taking calculated risks
📉
The Strategy Ranking Nobody Expected
  • Group discussion scored worst across all metrics — the default "let them talk it out" loses
  • Cognitive self-evolving planning won communication scores and lifted milestone achievement +3%
  • gpt-4o-mini topped research task score at 84.13 — but Minecraft (33.60) and Werewolf (36.33) stay brutally hard
  • Emergence has a cost: a Seer who never shared the werewolf's identity — information hoarding is emergent too
Interactive Demo — The Credit Ladder

One cooperative episode, graded two ways. Watch agents contribute turns; watch milestones unlock and credit land on the contributor. Then flip to binary scoring and watch all the signal vanish — the exact information milestone KPIs exist to preserve.

VERDICT
Collaboration quality is measurable — and mostly bad
The framework's finding cuts both ways: yes, you can now score coordination separately from task success; and no, current multi-agent systems do not coordinate well — group discussion, the field's default strategy, is its measured worst. The design grid it sits in comes straight from the collaboration-mechanisms survey; the competitive scenarios echo CAMEL's role-locked dynamics; and the planning-vs-acting split is Magentic-One's orchestrator pattern, factorized into an axis.
🪄 Emergent, and logged
Turn-level logs caught a Seer converting the village leader and a Witch saving her inspection potion for a critical vote — strategy nobody prompted, plus the hoarding failure nobody wanted.
🗣️ Communication as a scored axis
Separate 5-point C/P/C scores mean a team can win the task while failing to communicate — and the benchmark can say so, which binary scores never could.
🐺 Why Werewolf stays hard
Hidden roles require modeling other agents' beliefs and deception — Theory-of-Mind under adversarial uncertainty. The survey calls this model-based strategy; nobody implements it well yet.
🏅 Small-model surprise
gpt-4o-mini taking the top research task score (84.13) is the benchmark's quiet warning: coordination mechanics can matter more than raw model strength — the survey's thesis, quantified.
Test Yourself

Quick Quiz

Check your understanding of the key concepts from the MultiAgentBench paper.

Reference

Key Takeaways

Everything you need to remember about this paper.

✅ Six scenarios: four mutual-goal (research, Minecraft, database, coding) plus two conflicting-goal (Werewolf, bargaining) — 100 test cases each.
✅ MARBLE = Multi-agent cooRdination Backbone with LLM Engine — planners vs actors, 4 topologies (star/tree/graph/chain), 4 planning strategies.
✅ Metrics go beyond task success: milestone KPI for credit assignment, TS, and 5-point Communication, Planning, and Coordination scores.
✅ gpt-4o-mini averaged the highest Task Score (84.13 research TS); Minecraft (33.60) and Werewolf (36.33) remain brutally hard.
✅ Group discussion scored worst across all metrics; cognitive self-evolving planning won CS and lifted milestone achievement by 3%.
✅ Turn-level logs exposed emergent behavior — Seers turning leaders, Witches taking risks, and a Seer who never shared the werewolf's identity.