Early-layer TabPFN-v2 hidden states are low-rank under standard scalar tokenization, and RBF-expanded tokenization raises shallow-layer rank

- Wang et al. (2026) report that TabPFN-v2, whose tokenizer maps each scalar cell through a 1 x p linear projection, has low-rank hidden states, "especially early in the network": they run SVD on hidden states across the OpenML-CC18 benchmark and report numerical rank, Rank@99% and Rank@95% per layer (Sec. 3, Fig. 1), and truncating every attention/FFN module input to rank r barely changes TabPFN-v2's AUC (0.9177 at r = 192, 0.9143 at r = 50, 0.8985 at r = 20, 0.7674 at r = 5; Table 1). [wang-etal-2026-limix-2m-low-rank-collapse] - Proposition 3.1 proves that with a linear embedding and no positional embedding the embedded input has rank at most 2 (at most H + 1 after multi-head attention), and Appendix B argues theoretically that feature IDs/positional signals cannot add within-feature value degrees of freedom; these two statements are derivations, not measurements on TabPFN-v2. [wang-etal-2026-limix-2m-low-rank-collapse] - Replacing the linear projection with their RBF-expanded cell encoder (RaBEL) raises first-three-layer rank in a matched 2M-parameter comparison (numerical rank 58.41 to 78.62, Rank@99% 13.94 to 25.35, Rank@95% 6.73 to 12.31; Table 5); the resulting LimiX-2M and the 2M baseline are the authors' own models, and the paper reports LimiX-2M outperforming the larger TabPFN-v2 (7.24M) and TabICL (27.10M) on most benchmarks (Tables 3-6). [wang-etal-2026-limix-2m-low-rank-collapse]