TabICLv2 predicts by a literal class centroid; TabPFNv2 votes

measured in 1 paper

Bilos et al. find TabICLv2's prediction is reproduced almost exactly by a parameter-free per-class mean (centroid) readout (0.864 native vs 0.854 prototype-only) [bilos-etal-2026-mechanistic-study-tabular-foundation-models] TabPFNv2 and Mitra instead implement an attention-weighted vote, and replacing TabPFNv2's layer-9 attention with a uniform pattern collapses its accuracy from 0.87 to 0.49 [bilos-etal-2026-mechanistic-study-tabular-foundation-models] Transplanting the centroid rule onto the wrong backbone drops accuracy 33.1pp (onto TabPFNv2) and 9.0pp (onto Mitra), while transplanting the vote rule onto TabICLv2 drops it 39.5pp [bilos-etal-2026-mechanistic-study-tabular-foundation-models] Each backbone's own representation geometry is thus specifically required by its native readout rule [bilos-etal-2026-mechanistic-study-tabular-foundation-models] Zeroing TabPFNv2's positional weight matrix, or removing TabICLv2's RoPE, grants exact permutation invariance at no accuracy cost [bilos-etal-2026-mechanistic-study-tabular-foundation-models]

Structure

Linear Direction

Context

class-conditional mean/prototype readout as a literal geometric object, attention-weighted-vote readout as an alternative representational strategy, cross-backbone readout-rule transplantation causally confirming which geometry each model requires, positional-weight-matrix zeroing causally granting exact permutation invariance at no accuracy cost

Confirmed in models

TabPFN v2 (tabular in-context-learning foundation model) · TabPFN, Mitra · Mitra, TabICLv2 · TabICL

Method

Difference-in-means direction, Causal interventions (steering)

Papers

A Mechanistic Study of Tabular Foundation Models — Biloš, Marin, Wilson, James T., Schneider, Anderson, Nevmyvaka, Yuriy2026 · arXiv:2605.21288