Number-task subspaces are distinct yet linearly inter-transformable

measured in 1 paper

Hu, Niu & Varma analyze number-concept representations (comparison, parity, primality, successor/predecessor) across BERT, GPT-2, Qwen2.5-7B and Qwen2.5-Math-7B at ~75% depth [hu-etal-2026-the-representational-geometry-of-number] Task-specific subspaces are distinct via Procrustes disparity (~0.010 vs a permutation baseline 0.077-0.273) yet linearly inter-transformable (SVCCA canonical correlations ~0.80-0.90) [hu-etal-2026-the-representational-geometry-of-number] The parity axis sits nearly orthogonal to the magnitude axis (88.8 degrees) while the primality axis is only partly decoupled (68.8 degrees) [hu-etal-2026-the-representational-geometry-of-number] Subspace overlap between most task pairs stays below 0.2, except the closely-related successor/predecessor pair (0.36-0.42) [hu-etal-2026-the-representational-geometry-of-number]

Structure

Linear Subspace, Linear Direction

Context

number-representations, task-subspace-geometry

Confirmed in models

BERT-base-uncased · BERT, GPT-2 Small · GPT-2, Qwen-2.5-7B · Qwen, Qwen2.5-Math-7B · Qwen

Method

Generalized Procrustes Analysis (GPA), Singular Vector Canonical Correlation Analysis (SVCCA), Direction Extraction, Linear probing, Geometric analysis

Papers

The Representational Geometry of Number — Hu, Zhimin, Niu, Lanhao, Varma, Sashank2026 · arXiv:2602.06843