Agent Benchmark Paper Says Setup Choices Beat Bigger Models

The short version

A Munich led team tested coding agents on hard science tasks. They found that small setup choices often beat a bigger model or more run time. The work is a preprint on arXiv from 1 October 2026.

What the researchers did

Luis Wiedmann, Leander Girrbach, Cordelia Schmid, and Zeynep Akata work at Technical University of Munich, Helmholtz Munich, Inria, and ENS. They posted Agents Are Systems, Not Models: Rethinking Agentic Evaluation on arXiv. preprint, not yet peer-reviewed

They built a new test suite with four science jobs. Each job asks a coding agent to find a published specialist model and run it the right way. That is like a lab assistant who must pick the right tool from a huge catalog.

They changed five knobs on every run. Those knobs were task notes, reasoning mode, self checks, time limits, and the base model size. They logged more than 18,000 runs. They tracked success, cost, and how much scores jumped when they repeated the same setup.

They also shared the trajectories online so other labs can replay the runs. That openness helps teams test whether a vendor cherry picked one lucky path.

What they found

About 54% of the spread in scores came from luck on repeat runs. The same config could win once and fail once. That means a single demo can mislead buyers.

Clear task notes had the biggest lift. Notes beat extra time and beat a larger model on many tasks. Better notes also cut cost and made confidence scores more honest.

Extra time helped only when the agent already had enough context. It also helped when the base model was strong enough to use the time well.

Text that asked the agent to double check its work barely changed actions. A real verification tool changed actions a lot. Tools beat polite prompts.

The team also looked at how configs interact. Some pairs of knobs only worked when both were set well. That pattern fits real ops work where partial fixes fail.

Why it matters for you

If you run an AI platform team, you may get vendor slides that only compare model names. This paper says your runbooks and tool wiring may matter more. That shifts budget from raw tokens to engineering time.

If you run R and D ops, you should ask for repeat runs before you trust a leaderboard score. A shaky score could waste a quarter of pilot spend.

If you audit agents for risk, ask where verification lives. A prompt only check is weak if the system has no tool gate.

Procurement can add a simple clause. Vendors must publish repeat run variance, not only best case success. That one line could save months of rework.

What to keep in mind

The study is still a preprint. Peer review may tighten or change claims. The tasks focus on scientific coding agents. Your sales or support agent may behave differently.

Still, the core idea travels well. An agent is a system made of model, tools, prompts, and clocks. Buyers should score the whole bundle.

If you run evals in house, copy the paper habit of logging full trajectories. Text logs help you see where an agent wandered before it failed.

Key takeaways

• Run the same agent setup many times before you trust one score.
• Spend effort on task briefs and tool access, not only on model size.
• Put verification in tools, not only in friendly prompt text.
• Treat cost and calibration as first class metrics beside success rate.
• Ask vendors to show tool graphs, not only model cards on a slide.

When you read future agent benchmarks, check whether they treat the agent as frozen. If they do, the score may hide the knobs you will actually tune in production.

The authors argue that fair agent evals must treat configuration as part of the product. That view may slowly shift how labs report leaderboards.

Photo: ThisIsEngineering / Pexels

Photo: panumas nikhomkhai / Pexels

Sources

• Primary — arXiv, Agents Are Systems, Not Models: Rethinking Agentic Evaluation (1 October 2026). Benchmark design, variance splits, and configuration results.