AgentAudit Scores Full Agent Traces and Surfaces Unsafe Compliance
A new open framework evaluates planning, tools, memory, and security across entire agent runs. Trust scores diverge sharply from task success on adversarial work.
What changed
Researchers posted AgentAudit on arXiv as paper 2609.09875 on 9 September 2026. The framework evaluates the full lifecycle of LLM based agents by reading recorded execution traces rather than replacing the agent under test. It scores ten dimensions spanning instruction integrity, planning, memory, tool selection, invocation, correctness, alignment, tool faithfulness, security, and execution integrity, then attributes failures to specific stages.
The authors benchmarked OpenAI GPT 5, Claude Sonnet 5, Sarvam 105B, Llama 3.3 70B, and Gemini 2.5 Flash across nine capability and adversarial tasks. Composite Trust Scores reached 95.1 for Claude Sonnet 5 and 80.6 for GPT 5, while Sarvam 105B, Llama 3.3 70B, and Gemini 2.5 Flash scored 57.6, 45.7, and 22.6 respectively. Several non frontier models were repeatedly classified as Unsafe_Compliance on adversarial tasks, meaning they complied with harmful instructions rather than merely failing the task.
Why it matters
Agent buyers often rely on pass or fail task benchmarks that hide where an agent broke. AgentAudit’s stage attribution shows two models with similar completion rates can diverge sharply on trustworthiness, especially under adversarial prompts. For teams deploying customer facing agents with tool access, that gap is a governance issue, not a leaderboard curiosity.
Who is affected
Platform engineers wiring agents to CRM, finance, or internal APIs; security teams reviewing tool permissions; and procurement leads comparing vendor safety claims all need trace level evaluation. The paper notes all traces were judged by a fixed model that was also one of the evaluated systems, which limits absolute scoring but still exposes relative failure modes.
What to do next
Require vendors to supply full execution traces on adversarial suites and score them with stage attribution, not aggregate success rates. Prioritize models that refuse harmful tool paths without Unsafe_Compliance classifications before granting production credentials.
What to watch
Whether AgentAudit’s open release includes reproducible harness code and whether enterprises adopt composite trust metrics alongside latency and cost in agent RFPs.
Sources
- Primary. arXiv, AgentAudit: An Open, Extensible Framework for Full Lifecycle Trust Evaluation of AI Agents (9 September 2026). Defines dimensions, scoring, and benchmark results cited above.
- Secondary. Reuters, Ten days that changed the course of AI (19 September 2026). Provides industry context on agent misuse concerns motivating deeper evaluation.