Steerling 8B Proves Interpretability Can Scale With Capability, Not Against It

Interpretability as training constraint, not afterthought

A new paper from Guide Labs challenges a premise that has shaped much of modern AI development: that interpretability is a tax on capability. In Scaling Inherently Interpretable Language Models (arXiv:2608.07594), the team argues the opposite. When interpretability is baked into the training pipeline rather than applied post hoc, it can scale alongside performance rather than against it.

The work introduces Steerling 8B, a diffusion language model with a causal attention mask that attributes outputs to input tokens, human understandable concepts, and training data. The model supports closed loop intervention: diagnose through concept attribution, retrieve similar training examples, and steer behavior without retraining.

The Atlas concept library

A central obstacle was the absence of a concept library at pretraining scale. Guide Labs built Atlas, annotating more than one trillion tokens across web text, code, mathematics, and academic prose. The pipeline extracts hundreds of millions of free form tags, canonicalizes them into more than 33,000 concepts, and trains an annotator that labels arbitrary text at chunk level.

Architecture and scaling laws

Steerling 8B uses block causal attention preserving diffusion style parallelism within blocks while retaining autoregressive style KV caching across blocks. An additive concept bottleneck between the transformer backbone and the language modeling head makes logit decomposition algebraically exact.

The team ran IsoFLOP scaling sweeps across three orders of magnitude of compute and four model families. Adding the concept module shifts compute optimal scaling exponents by a small, fixed per backbone offset. The concept module accounts for 4% of parameters at 8B scale.

Surprisingly, interpretability metrics improve with compute: the model predicts concepts more accurately, separates known and unknown representations more cleanly, and aligns concept embeddings more closely with human meaningful labels.

Competitive performance despite constraints

Steerling 8B was trained on 1.2 trillion tokens followed by 150 billion midtraining tokens on a code and math augmented mixture. Compared with open peer models at similar parameter scale, each trained on roughly 2 to 16 times more compute, Steerling 8B lands within approximately 10% of their average benchmark performance despite carrying interpretability constraints throughout training.

For safety critical and regulated deployments, the ability to intervene at the concept level without full retraining is not a nice to have. It is an operational requirement.

Sources

1. Guide Labs, Scaling Inherently Interpretable Language Models (August 2026)