MATH · IN · MODELS

Mamba

CMU / Princeton (Gu & Dao)

Structures found in this family (4)

Linear Direction, Linear Subspace, Circle, Anisotropy

By model (5)

Mamba-2.8B · 2.8B
Linear Subspace, Circle, Anisotropy
Mamba-1.4B · 1.4B
Anisotropy
Mamba-130m (state-spaces/mamba-130m-hf) · 130M
Anisotropy
Mamba-370M (state-spaces/mamba-370m) · 370M
Linear Direction
Mamba-8B · 8B
Linear Direction

Observations (6)

In a real pretrained Mamba-370M state-space model's Indirect Object Identification circuit, name-entity information appears linearly represented at Layer 39's SSM input (averaged-activation replace / subtract-and-add edits overwrite the output name), after Layer 39's convolution shifts the relevant name one position forward
Fourier periodicity in number embeddings is universal; linear separability is not
In-context task representations are k-NN-decodable and causally patchable across real LLMs and an SSM
Knowledge editing shatters transformers' natural cyclic manifold geometry
Mamba SSMs distribute representational variance evenly across dimensions whereas transformers are dominated by a single principal direction (increasingly skewed after each attention layer in a hybrid), yet both encode concepts in rank-constrained subspaces of similar dimensionality and have highly aligned local semantic manifolds
Frozen Mamba collapses into an extreme anisotropy cone

Papers

Investigating the Indirect Object Identification circuit in Mamba (2024), Convergent Evolution: How Different Language Models Learn Similar Number Representations (2026), Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder-Decoder Perspective (2025), Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing (2024), Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers (2026), Lost in State Space: Probing Frozen Mamba Representations (2026)