The Drifting Target: Why LLMs Lose the Plot When User Intent Evolves Mid-Conversation

Benchmark leaders have spent years optimizing models for tasks where the user's goal is fully specified upfront. A paper posted to arXiv on July 22, 2026 argues that setup is quietly wrong — and that the gap it hides may matter more than another point on a static leaderboard.

The static-setting illusion

In "LLMs Get Lost in Evolving User Intent" (arXiv:2607.20734), Jihoon Tack, Philippe Laban, and Jennifer Neville introduce a framework that transforms single-turn benchmarks into multi-turn conversations where user intent evolves across turns — incrementally revealed, revised, and sometimes redirected mid-conversation — while preserving each task's original evaluation protocol.

The result is uncomfortable: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families.

That matters because the industry’s near-term product shape is collaborative agents taking on delegated tasks through iterative interaction — not one-shot Q&A.

Why existing benchmarks miss the point

Most LLMs are still evaluated or trained in single-turn, fully-specified settings. Users rarely behave that way. They disclose requirements late, change constraints after seeing a draft, or pivot when the model's first answer reveals a misunderstanding.

The authors' framework lets existing benchmarks be reused as controlled testbeds without new annotation — a practical design choice that makes the failure mode hard to dismiss as an artifact of a bespoke dataset.

What the authors conclude

The paper's bottom line is blunt: today's LLMs do not yet faithfully track and act on evolving user intent — a capability invisible to static evaluation yet critical for collaborative agents.

For labs shipping agent products in 2026, the implication is methodological, not cosmetic. A model that tops a single-turn coding or reasoning benchmark may still lose users the moment a real conversation starts moving.

Connection to the deployment stack

This research lands alongside Google DeepMind's $10 million multi-agent safety funding call and Anthropic's zero-trust agent deployment guidance — separate problems, same theme: interaction is the new unit of analysis.

The authors do not claim evolving-intent evaluation is solved — only that ignoring it leaves a hole in the evidence base large enough to mis-rank today's frontier models.

### Sources

• arXiv — LLMs Get Lost in Evolving User Intent (July 22, 2026)
• MIT Technology Review — Google DeepMind is worried about what happens when millions of agents start to interact (June 11, 2026)