Agent teams cost up to 5x more in Vals AI's test

Before you spin up a team of agents, try turning up the effort on one. It is often cheaper and about as good.

Vals AI built the same 50 web apps eight different ways and published the results on Oct. 9. A lead agent with up to five subagents cost 1.8 to 5.1 times as much as a single agent. Only one of four team setups scored significantly better.

One win in four tests

The test was Vals' Vibe Code Bench, where a model builds a full web app from a product spec and is graded on the share of UI tests it passes. Vals ran OpenAI's GPT 6 Sol and Anthropic's Claude Opus 5.5, each at medium and max reasoning effort, each as a single agent and as a team.

The one clear gain: GPT 6 Sol at medium effort went from 77.6% alone to 84.9% as a team, a 7.3 point jump (p = 0.005). Its cost per app went from $1.22 to $3.07.

The other three differences were not statistically significant. Opus 5.5 at medium effort scored 91.5% alone and 91.2% as a team, at more than twice the cost.

At the top end, Opus 5.5 at max effort as a team scored 93.2% and cost $122 per app. The single Opus agent at medium effort scored 91.5% for $4.08.

About 30 times the money for under two points.

Effort beat headcount for one model

For GPT 6 Sol, raising the reasoning effort on a single agent gained 11.4 points (p < 0.001), from 77.6% to 89.0%, for $4.82 per app. That is a bigger jump than adding a team at medium effort.

For Opus 5.5, max effort alone did not move the score significantly (p = 0.48). So the lesson is not universal. It depends on the model.

Vals' own summary is careful: "a team can boost performance, but higher reasoning effort is an alternative worth weighing."

The caveats are real

Vals ran each of the eight setups once, so the p-values reflect variation across the 50 apps, not repeated runs. The two models used different harnesses, so comparing them mixes the model with the agent harness. Teams also got a delegation prompt the single agents did not.

And Vals warns the results "may not carry over to other kinds of work." This is one benchmark of full-stack web apps.

Granted. It is still a public, itemized cost comparison of agent teams against a single agent, run on two current frontier models.

What it means for your bill

If you build with coding agents, you have heard the multi-agent pitch. This is a number to put next to it.

Run your own version before you commit. Take ten real tasks. Run one agent at medium effort, one at high, and a team. Compare the pass rate and the invoice.

If the team does not beat the higher-effort single agent by a clear margin, you are paying for parallelism you do not need. Watch tokens as closely as scores. Vals found Opus max-effort subagents read a median of 224 million cached tokens per app, against 55 million for the single agent.

Teams were not useless. GPT 6 Sol's medium team gained about 15 points on the harder apps, Vals says, and about 1 point on the easier ones. If your work is hard, a team may earn its keep.

What to watch

Whether other evaluators repeat this with more runs and other kinds of tasks. Whether the agent vendors publish cost per task alongside their scores.

Until they do, measure it yourself. Then pick the cheaper setup that passes.