AgentSuite Paper Audits Agent Benchmarks Before Leaderboards

The short version

A new study from Georgia Tech and NVIDIA says many agent leaderboards mix model mistakes with broken test tasks. The team built a checker called COBA that splits each task into four parts and hunts flaws before scores are trusted. The work appeared at ICML 2026 and ships as open software called AgentSuite.

What the researchers did

Hyewon Suh, Binfei Ji, Seojune Lee, and colleagues at Georgia Institute of Technology worked with NVIDIA and UC Santa Barbara researchers. They published AgentSuite: Toward More Reliable Agent Evaluation with a Component-Based Benchmark Auditing Pipeline in the ICML 2026 proceedings (PMLR volume 306). The paper introduces COBA, which stands for component based benchmark auditing.

COBA treats every agent benchmark as four linked pieces: the user prompt, the environment, the ground truth answer path, and the scoring rules. Rule based scripts catch hard errors. A guided language model pass catches softer semantic problems. The team tested COBA on six widely used agent benchmarks and compared its flags to expert human reviews.

Photo: ThisIsEngineering / Pexels

What they found

The authors report F1 scores between 0.791 and 0.874 when COBA’s labels are matched against expert judgments across the six benchmarks. That range means the automated auditor agrees with skilled reviewers most of the time. They also say COBA found issues on complex suites such as τ²-bench that manual review missed because the tasks are large and tangled.

Their analysis concludes that benchmark flaws are common and that those flaws change which model looks best. When a task script is wrong, a strong model can look weak and a weak model can look strong. COBA is meant to filter or fix those tasks before anyone trusts a leaderboard rank.

Photo: fauxels / Pexels

Why it matters for you

If you buy or build tool using agents, your eval stack is part of the product. A score on a noisy benchmark can send you toward the wrong vendor or the wrong internal design. COBA style auditing gives ops and research leads a way to ask whether a bad run was the model or the test.

Product teams can treat audited benchmark variants as a cleaner regression suite before launch. Investors and policy staff get a clearer read when two labs claim the same leaderboard spot. The open AgentSuite release is meant to run those audits without rebuilding the whole harness from scratch.

Save the ICML PDF link in your eval wiki. When a vendor cites AgentSuite scores, ask whether they ran COBA cleaned tasks or the raw benchmark. Replay one audited task locally before you switch models in production.

What to keep in mind

This is a conference paper with open code, not a hosted certification service. COBA still uses model based checks for some classes of flaw, so its own error rate must be watched on your domain. Auditing takes effort, and fixing benchmarks can break comparability with older published numbers.

Teams should not assume one pass makes a benchmark perfect forever. Tasks drift when tools and APIs change. Re audit after major environment updates.

What a careful reader should do next

Pick one agent benchmark your team already runs. Skim COBA issue types and check whether your harness logs enough detail to audit. Fix one flagged task before the next model bake off.

Key takeaways

• Agent benchmarks can hide bugs in prompts, worlds, answers, or graders.
• COBA splits tasks into four components and flags issues with rules plus model review.
• Reported F1 against experts falls between about 0.79 and 0.87 on six public suites.
• Flaws can swap model rankings, so audits matter before big bets.
• AgentSuite on GitHub packages COBA with runnable benchmark variants.

Sources

• Primary — Proceedings of Machine Learning Research, AgentSuite: Toward More Reliable Agent Evaluation with a Component-Based Benchmark Auditing Pipeline (ICML 2026). Defines COBA, reports F1 alignment, and describes AgentSuite.
• Secondary — GitHub, Agent-Suite/AgentSuite (2026). Open platform named in the paper for running audits and benchmarks.