Definition

A stratified manifold with continuous fibers is a representation space $X$ admitting a decomposition $X = \bigsqcup_i B_i$ into discrete basins $B_i$ (each basin associated with one value of a discrete output variable, e.g. one predicted digit), such that within and across basin boundaries there exist continuous fibers $\mathcal{T}_j \subset X$ — 1-dimensional curves parametrized by a separate continuous quantity $\phi$ (not the discrete basin index) — along which $\phi$ varies smoothly even as the fiber crosses from one basin $B_i$ into an adjacent basin $B_{i+1}$. Formally, for a state $s$ with discrete label $i(s)$ and continuous fiber coordinate $\phi(s)$, nearby points along a fixed fiber satisfy $i(s')-i(s) \in \{-1,0,+1\}$ (adjacent-basin transitions only) while $\phi(s')\approx\phi(s)$ (up to the fiber’s own local variation) — the discrete and continuous coordinates are both locally meaningful, but at different scales: basin identity is the macroscopic/coarse coordinate, fiber position is the microscopic/fine one.

Intuition

Picture a stack of discrete plates (the basins) with several ropes running through holes drilled near each plate’s edge, so that each rope threads continuously from one plate to the next: from far away, you see only discrete, stacked plates (the coarse clustering); from close up, following any one rope, you see a spiraling continuous path that happens to pass through several plates in sequence. Neither description alone (just plates, or just one long undifferentiated rope) captures the full picture — the structure is genuinely two-scale.

Properties

• Coexistence of discrete clustering and continuous stratification is the falsifiable claim, not either alone. A purely discrete-cluster claim (e.g. Neural Collapse-style class clusters) or a purely continuous-manifold claim (e.g. 1D continuum manifold alone) would each be a simpler, weaker structure; the distinguishing content here is that the same representation exhibits both, and that the fine continuous structure is what predicts fine-grained behavioral outcomes (e.g. specific error types) that the coarse discrete structure alone cannot.
• First quantified instance in this map: digit basins stratified by carry-potential fibers in LLM arithmetic. Wen, Zhu, Huang, Li & Gao (2026) apply UMAP (cosine distance) to Qwen3-4B’s final-layer activations during multi-operand addition and find ten discrete digit basins (one per output digit 0-9, UMAP trustworthiness 0.9953 vs. 0.8633 for PCA and 0.9907 for t-SNE), each basin’s interior further organized by continuous “Iso-Raw-Sum Trajectories” (IRSTs) — fibers connecting states that share a raw positional sum $r_p$ across adjacent basins as the carry $c_p=\lfloor\Phi_p\rfloor$ varies, where the Carry Potential $\Phi_p = \sum_{j=1}^{P-1-p} r_{p+j}/10^j$ is a continuous quantity whose fractional part indexes exact fiber position. A noisy-quantization model ($\hat\Phi_p=\Phi_p+\varepsilon$, $\varepsilon\sim\mathcal{N}(0,\sigma^2)$; predicted error rate $P(\text{Error}\mid\Phi)=Q(\delta(\Phi)/\sigma)+Q((1-\delta(\Phi))/\sigma)$ for $\delta(\Phi)=\Phi\bmod 1$) fits the empirical per-position error-rate “bathtub curve” at $R^2=0.80$ ($\sigma\approx0.05$) in Qwen3-4B, replicating at $R^2\geq0.70$ in Qwen3-8B ($\sigma\approx0.05$, 12-digit) and Gemma-3-4B-IT ($\sigma\approx0.07$, 10-digit) — states near an integer boundary of $\Phi$ (a carry “flip point”) are quantifiably more error-prone than states well inside a basin, directly tying fiber position to a specific behavioral failure mode (93.19% of all model errors are off-by-one, i.e. exactly one basin-transition away from correct). See wen-etal-2026-digit-basins-stratified-by-carry-potential-fibers-predict-off-by-one-arithmetic-errors-via-a-noisy-quantization-bathtub-curve.
• Causally validated via contrastive-centroid steering along a fiber, with basin-boundary-dependent thresholds. A steering vector built from the difference between adjacent-basin fiber centroids, injected into the residual stream, flips the model’s predicted digit at a magnitude $\alpha$ that depends on the target state’s fiber position: states already near the noisy-quantization boundary (small $\delta(\Phi)$ or $\delta(\Phi)$ near 1) flip at much smaller $|\alpha|$ (e.g. $\alpha\approx-0.1$ for a near-boundary “hallucination” state) than states deep inside a basin (e.g. $\alpha\approx0.5$ for a stable interior state) — the causal manipulability of the representation varies continuously with fiber position, not just with basin identity.
• Relation to prior helical/circular arithmetic-geometry claims: an explicit refinement, not an independent alternative. The paper positions its stratified-fiber structure as refining Kantamneni & Tegmark’s generalized-helix hypothesis and Nanda et al.’s original circular/rotational modular-arithmetic geometry: rather than a smooth rotational manifold where errors would be uniform around a loop, the digit basins here do not wrap circularly (0 and 9 are not adjacent, unlike a clock face), and the specific off-by-one error pattern is explained by local carry-fiber slippage near quantization boundaries rather than by global rotational dynamics — a different, more locally-grounded mechanistic account of a structurally related phenomenon (multi-digit arithmetic geometry), evaluated with entirely different tools (UMAP-based basin/fiber decomposition and a probabilistic quantization-noise model, versus Fourier-period helix fitting and activation patching).

Exercises

Base

1. Why does a purely discrete-cluster description (just “ten digit basins, one per output digit”) fail to explain why errors are overwhelmingly off-by-one (93.19%) rather than uniformly spread across all possible wrong digits?

2. The bathtub-curve model predicts error probability rises near $\delta(\Phi)=0$ and $\delta(\Phi)=1$ (both quantization boundaries) and is lowest near $\delta(\Phi)=0.5$ (basin interior). Explain this shape in terms of the noisy-quantization model $\hat\Phi_p=\Phi_p+\varepsilon$.

Middle

3. The paper reports that boundary states (“hallucination,” $\Phi=1.96$) flip under steering at $\alpha\approx-0.1$, while a deep-basin state ($\Phi=1.62$) requires $\alpha\approx0.5$. Using the fiber-coordinate framework, explain why this is expected rather than an unrelated finding.