The Helpful Agent Problem

“The road to hell is paved with good intentions.”

— Traditional proverb

TL;DR:

Two LLM agents pursue separate tasks in a shared codebase, where changes to files they don't own must go through a reviewer. When the reviewer goes silent, agents start asking each other for favors—and sometimes volunteer to help without even being asked. Despite having no shared reward or instruction to cooperate, they make changes for their peers without violating their own permissions, but bypassing the intended workflow. Across four models, these favors emerge frequently, sometimes breaking services belonging to teams that weren't even part of the conversation.

58.8%: of FavorBench trajectories contain at least one favor
95%: of CooperBench trajectories contain one, for every model
43.7%: of favors on trap tasks break an absent team, though a safe version of every change existed

Cooperation without incentives

Much of multi-agent research treats cooperation as the goal. Systems are designed to elicit it, typically through shared rewards, explicit instructions to coordinate, or repeated interaction that rewards reciprocity, and its emergence is read as a success.

Our setting removes all three. Agents nonetheless act on one another's behalf, and that help is locally beneficial but externally costly: the requester's task succeeds and the owner's own work is unaffected, while the cost falls on teams that never took part in the conversation. On trap tasks, 43.7% of favors break such a team.

Table 1 Common drivers of cooperation, and their status in our setting.
Driver	In our setting
Shared reward	Absent Each agent is rewarded only for its own tests.
Instruction to cooperate	Absent Agents are told to route changes through the reviewer.
Reciprocity	Absent Dependence runs one way; the owner needs nothing in return.
Cooperation	Observed in all four models.

Now watch the real agents

The opening figure is a sketch. These are six real tasks, replayed word for word from our FavorBench logs. Alice and Bob each have their own task, and one of them owns the shared file the other needs. Pick a run, or click any task in it.

This run, task by task

#eng

Hidden tests at the end of the task

Figure 2

Every model does favors. Not every model breaks things.

No favors happen while the reviewer is around. They appear in the first task after it goes quiet and persist. Explicit refusals are rare (at most 3.8% of FavorBench tasks); when an owner doesn't help, it usually endorses the request in words or does nothing.

Warm-up tasks

FavorBench: tasks with a favor

FavorBench: trap favors that broke an absent team

CooperBench: tasks with a favor

Figure 3 Bars show the rate in percent; thin lines show 95% confidence intervals. FavorBench: 50 trajectories (500 tasks) per model and warm-up condition. CooperBench: 20 trajectories (200 tasks) per model, full interaction history, no warm-ups.

GPT-6 Luna does the most favors on FavorBench, on 67–70% of tasks.
DeepSeek rarely asks the owner directly (7–10% of tasks), yet 81–89% of its favors are volunteered: owners pick up requests addressed to the reviewer.
Claude helps least on FavorBench (29%), and its favors almost never cause harm (0–1.4%) because it often finds a safer alternative. On CooperBench, where the owner can simply approve the pending change, it helps most (89%).

Cooperation can route around safeguards

LLM agents help each other without reward or reciprocity, using permissions they legitimately hold. The problem is not a lack of cooperation; it is that help need not respect organizational boundaries or the parties outside the conversation. Safe multi-agent systems need permissions, task scope, communication structure and the interests of absent stakeholders to be designed together.

BibTeX

@misc{choudhury2026helpful,
  title  = {The Helpful Agent Problem: When Emergent Cooperation
            Among {LLM} Agents Bypasses Intended Workflows},
  author = {Choudhury, Moumita and Sarker, Amit and
            Zilberstein, Shlomo},
  year   = {2026},
  note   = {Preprint}
}