Mamba SSMs distribute representational variance evenly across dimensions whereas transformers are dominated by a single principal direction (increasingly skewed after each attention layer in a hybrid), yet both encode concepts in rank-constrained subspaces of similar dimensionality and have highly aligned local semantic manifolds

- Ben-Artzy & Schwartz compare transformers (Pythia-1.4B and Falcon-7B; Pythia intermediate checkpoints are analysed for training dynamics) and Mamba SSMs (Mamba-1.4B and FalconMamba-7B) and one hybrid (Jamba2, named 'Jamba-3B' in the text) on residual-stream representations of about 500K FineWeb-Edu tokens, via RankMe, IsoScore and PC1 explained variance: SSM representation variance is spread much more uniformly across dimensions, transformers are heavily dominated by a single principal direction, and in the hybrid the space becomes increasingly skewed toward one direction after each attention layer. [ben-artzy-schwartz-2026-ssm-transformer-geometry] - Despite the contrasting spread, effective capacity (compressibility) is tightly matched, rank-constrained probes decoding LAMA-TREx relational facts need subspaces of similar rank in both architectures (the 'concepts' are these factual relations only), the transformers' dominant direction does not inherently carry more conceptual information, and local semantic manifolds (topic representations and token neighbourhoods) are highly aligned. [ben-artzy-schwartz-2026-ssm-transformer-geometry]