Chain-of-thought traverses step-specific subspaces that sharpen with depth

measured in 1 paper

Sun et al. train per-layer per-step one-vs-rest logistic probes on activations preceding each reasoning-step marker in Llama-3.1-8B Base, Instruct, and R1-Distill [sun-etal-2026-llm-reasoning-as-trajectories] Later-step separability starts low at the embedding layer and exceeds 0.90 only after several layers, while early steps are separable from layer 0; shuffling controls collapse to about 0.59 [sun-etal-2026-llm-reasoning-as-trajectories] Probes transfer across training regimes, prompt formats, and datasets (0.83-1.00), and late-trajectory PCA features predict eventual correctness mid-generation (ROC-AUC up to 0.87) [sun-etal-2026-llm-reasoning-as-trajectories] A termination-direction steering vector causally controls reasoning length, and a rank-32 trajectory-correction update raises 6-7-step GSM8K accuracy by about 7.6 points while preserving correct answers [sun-etal-2026-llm-reasoning-as-trajectories]

Structure

Linear Separability

Context

step-specific subspaces identified via per-layer one-vs-rest logistic probes, separability increasing with depth, near-ceiling for early steps and requiring depth for later steps, step-specific structure transfers across training regimes, prompt formats, and datasets, reasoning training reshapes depth-of-convergence to a termination subspace rather than creating new structure, late-trajectory PCA features predict mid-reasoning correctness (ROC-AUC up to 0.87), additive termination-direction steering causally controls reasoning length, low-rank trajectory-correction steering causally improves multi-step reasoning accuracy

Confirmed in models

Llama-3.1-8B · Llama, Llama-3.1-8B-Instruct · Llama, DeepSeek-R1-Distill-Llama-8B · DeepSeek

Method

Linear probing, Causal interventions (steering)

Papers

LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals — Sun, Lihao, Dong, Hang, Qiao, Bo, Lin, Qingwei, Zhang, Dongmei, Rajmohan, Saravan2026 · arXiv:2604.05655