TabPFN linearly encodes regression coefficients and arithmetic intermediates

measured in 1 paper

Gupta, Sethi & Kumar probe TabPFN v2 on synthetic data with known form: for z=alpha*x+beta*y linear probes recover the coefficients with high R^2 (sharp rise at layer 6), and for z=a*b+c a linear probe recovers the intermediate product a*b, concentrated in middle layers [gupta-sethi-kumar-2026-tabpfn-looking-glass] A probe-complexity sweep (linear to deep MLP) shows R^2 decreases monotonically with added complexity, the standard signature of a genuinely linear encoding [gupta-sethi-kumar-2026-tabpfn-looking-glass] A logit-lens vs linear-probe gap (answer decodable at layer 5 but native-space-aligned only by 7-8) is read as computational overthinking [gupta-sethi-kumar-2026-tabpfn-looking-glass] No causal intervention is performed; this is the first such linear-vs-complexity analysis for a tabular foundation model [gupta-sethi-kumar-2026-tabpfn-looking-glass]

Structure

Linear Subspace

Context

linear-probing of regression coefficients (alpha, beta) from ICL activations, linear-probing of an arithmetic intermediate term (a*b) mirroring order of operations, probe-complexity sweep (linear vs. MLP) as evidence for a genuinely linear encoding, logit lens vs. linear-probe layer-of-emergence gap, dataset-specific vs. universal coefficient encoding (cross-fit pooling fails; within-fit succeeds)

Confirmed in models

TabPFN v2 (tabular in-context-learning foundation model) · TabPFN

Method

Linear probing, Mlp probing, Logit Lens

Papers

TabPFN Through The Looking Glass: An Interpretability Study of TabPFN and Its Internal Representations — Gupta, Aviral, Sethi, Armaan, Kumar, Dhruv2026 · arXiv:2601.08181