Falcon-Mamba

TII

Structures found in this family (2)

Linear Subspace, Anisotropy

By model (1)

Falcon-Mamba-7B · 7B

Linear Subspace, Anisotropy

Observations (2)

Fourier periodicity in number embeddings is universal; linear separability is not

Mamba SSMs distribute representational variance evenly across dimensions whereas transformers are dominated by a single principal direction (increasingly skewed after each attention layer in a hybrid), yet both encode concepts in rank-constrained subspaces of similar dimensionality and have highly aligned local semantic manifolds

Papers

Convergent Evolution: How Different Language Models Learn Similar Number Representations (2026), Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers (2026)