Falcon

TII

Structures found in this family (2)

Linear Subspace, Anisotropy

By model (2)

Falcon-7B-Instruct · 7B

Linear Subspace, Anisotropy

Falcon-7B · 7B

Anisotropy

Observations (3)

A low-dimensional subspace of decode-time hidden states is shared across tasks in LLMs, and removing it only during decoding degrades decisions far more than removing a prefill-derived or random subspace

Mamba SSMs distribute representational variance evenly across dimensions whereas transformers are dominated by a single principal direction (increasingly skewed after each attention layer in a hybrid), yet both encode concepts in rank-constrained subspaces of similar dimensionality and have highly aligned local semantic manifolds

Decoder-only models peak anisotropy in middle layers

Papers

DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions (2026), Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers (2026), The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models (2024)