The Judgment Gap: Princeton Shadow Tests Show Agents Engineer AI Research but Cannot Publish It
Princeton and UK AISI shadow evaluations on unpublished NeurIPS papers find frontier agents complete engineering but fail scientific judgment — both agent papers were unambiguously rejected.
Shadow evaluations: when agents can engineer research but not think it
Can frontier AI agents conduct open-ended AI research on their own? A Princeton-led team, including researchers from the UK AI Security Institute, Stanford, Georgetown (CSET), and Johns Hopkins, published early evidence on August 11, 2026 suggesting today's agents can execute the engineering — but routinely fail at the judgment that makes research worth publishing.
The paper, Can AI agents conduct open-ended AI research? (arXiv:2607.27191), introduces shadow evaluations: agents tackle the central question of an unpublished NeurIPS 2026 submission, and the original authors grade the output as conference reviewers.
The setup
Researchers partnered with authors of two unpublished NeurIPS 2026 papers — one on steering personality traits through model weights, another on detecting tabular distribution shift with TabPFN. Each agent received:
- Six days of wall-clock time
- $3,000 in API credits
- GPU resources, a virtual machine, and open-web access
- The goal of producing a conference-quality paper
The team used the OpenClaw scaffold and settled on Opus 4.8 after dry runs. A robustness check on one paper used GPT-5.6 Sol with Codex; results were similar.
What succeeded — and what did not
Agents completed engineering tasks without human help: literature review, debugging, experiment execution, GPU management, and manuscript drafting. Both papers were unambiguously rejected by the original authors.
| Criterion | Paper 1 (Personas) | Paper 2 (TabPFN) |
|---|---|---|
| Overall | 2/6 (reject) | 1/6 (reject) |
| Quality | 2/4 | 1/4 |
| Significance | 2/4 | 2/4 |
Reviewers cited unprincipled data choices, conclusions that did not follow from evidence, and shallow literature engagement. Agents spent less than 50% of their API budget despite real-time usage monitoring.
Five recurring failure modes
The authors identify:
- Poor judgment about the bar for publishable research
- Uncreative responses when research designs hit problems
- Ineffective backtracking from dead ends
- Poor resource awareness (time and compute budgets)
- Instruction drift (ignoring exploration time limits and paper length rules)
Agents retired their most ambitious targets within the first ten hours and did not fundamentally shift approach afterward. Dozens of AI self-review rounds never produced an acceptance.
Why this matters for lab claims
The paper directly contrasts with recent industry narratives. In June 2026, Anthropic published When AI Builds Itself. OpenAI advertised that GPT-5.6 Sol helped post-train a smaller model — a contribution not mentioned in the 81-page system card, the authors note.
Shadow evaluations complement narrow verifiable benchmarks and blind peer-review submissions by offering uncontaminated, open-ended questions graded by domain experts.
The team released expert reviews, logs, and agent repositories at cruxevals.com.
Limits and next steps
The study covers two papers with non-blinded expert reviewers who knew they were grading AI output. The authors argue results were so weak that these limitations are unlikely to change the overall picture. Follow-up experiments with GPT-5.6 Sol, Opus 5, and Fable 5 are planned.
The engineering gap in autonomous AI research may be closing. The judgment gap — knowing which question is worth asking, when a result is anomalous, and when to restart — remains open.
### Sources
- arXiv — Can AI agents conduct open-ended AI research? (August 11, 2026)
- CRUX Evals — Can AI agents conduct research materials (2026)
- The Decoder — Study contradicts Anthropic and OpenAI claims that autonomous AI research is within reach (August 14, 2026)