A conceptual subspace emerges in the middle-to-late layers of base LLMs (Llama-3.1 70B and 8B, Llama-3 8B, Qwen2.5 3B-32B) during in-context inference, stays stable across contexts, and is causally central to predictions (restoring it recovers most performance, ablating it degrades performance); the effect is strongest in the 70B model

- Xu et al. apply SVD and generalized canonical correlation analysis (GCCA) to final-token hidden states of base LLMs during in-context reverse-dictionary inference (THINGS concepts; four further tasks from Todd et al. in the appendix) and find a shared conceptual subspace emerging in middle to late layers (GCCA subspace dimension about 1,180 in Llama-3.1 70B with hidden size 8192, about 1,250 in Llama-3.1 8B with hidden size 4096) whose structure persists across contexts and becomes more coherent with more demonstrations; its relational structure is preserved under cross-context subspace transfer. [xu-etal-2026-emergent-structured-representations-icl] - Causal mediation in Llama-3.1 70B shows the subspace is functionally central: patching it into middle layers recovers about 91% (description corruption) and 88% (label corruption) of performance at layer ~40 and 98% under query corruption, ablating it degrades prediction without collapsing it (partial redundancy), and isolating it keeps near-original performance across most layers; the effect is more pronounced in larger models (smaller models rely on more diffuse representations, and in Qwen2.5-3B random subspaces often match the derived one). The study also reports that early-to-middle-layer heads construct the subspace and later layers use it. [xu-etal-2026-emergent-structured-representations-icl]