Dependencies, Not Documents: OctoLong Retrains Long-Context Models on Cross-Repository Code Graphs

Long-context language models are table stakes for agentic coding — but most training corpora treat repositories as static snapshots. A paper posted to arXiv on August 5, 2026 argues that the missing ingredient is cross-repository dependency structure, and introduces OctoLong, a pipeline that instruments an AST parser, language server, and package manager to recursively retrieve code references and build contexts spanning millions of tokens.

The claim

Indraneil Paul, Falko Helm, Goran Glavaš, and Iryna Gurevych (UKP Lab, TU Darmstadt) present OctoLong-Instruct, a suite of open models from 600M to 14B parameters trained via context-extension mid-training on a ~50B-token mixture containing ~6.2B tokens of OctoLong code contexts, followed by ~10B tokens of instruction tuning.

Their headline result: replacing just 12% of traditional context-extension corpora with OctoLong data yields substantial gains in long-range retrieval, long-term state tracking, repository-level code understanding, and downstream agentic tasks — while also improving API usage in short-context coding scenarios.

Why books and repos fall short

Existing long-context corpora — books, academic articles, single-repo snapshots — are finite and often lack long-distance dependencies. Real software is different: imports chain across packages, types resolve through language servers, and refactors propagate through dependency graphs.

OctoLong's pipeline:

1. AST parsing to locate references.
2. Language server backend for semantic resolution.
3. Package manager integration for recursive retrieval across repositories.

The output is dependency-rich code contexts that mirror how engineers actually navigate large systems — not flattened file dumps.

Where gains are real

The authors benchmark against 18 state-of-the-art open-weight long-context LMs. Improvements appear across:

• Long-range retrieval within massive codebases.
• Long-term state tracking across multi-file edits.
• Repository-level understanding (architecture, not just syntax).
• Downstream agentic tasks requiring tool use over extended horizons.
• Short-context API usage — a secondary benefit suggesting the mid-training signal generalizes.

Limitations and measurement checklist

Before treating OctoLong as a universal fix, verify:

• Language coverage: The pipeline depends on AST + LSP + package manager support per ecosystem.
• Compute budget: ~50B-token mid-training plus ~10B instruction tuning is not cheap at 14B scale.
• Evaluation protocol: Gains are reported against open-weight baselines under the authors' matched protocol — cross-check on your own repos.
• Dependency freshness: Cross-repo contexts go stale when upstream packages change; the paper does not address continuous re-indexing.

Pragmatic agenda for labs

1. Treat context engineering as infrastructure, not a one-time corpus build.
2. Instrument LSP-aware retrieval before scaling raw token count.
3. Measure agentic benchmarks, not just perplexity on long documents.
4. Open-weight replication: base models 600M–14B are specified; Ascend NPU training details may affect reproducibility on other hardware.

Takeaway

OctoLong reframes long-context training for code: the bottleneck is not token length alone but dependency topology. Twelve percent of the right cross-repo signal may outperform doubling generic text — a useful reminder as agentic workflows push context windows toward seven and eight figures.

### Sources

• arXiv — OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling (August 5, 2026)
• Paul et al. — DOI 10.48550/arXiv.2608.05141 (2026)