Jacobian centroids (row-sums of a sub-component's local input-output Jacobian) form cleaner, more transferable linear feature directions than raw activations across four real-model settings: ResNet50/DTD, DINOv2-vs-DINOv3 SAE dictionaries, GPT2-Large circuit discovery, and Llama-3.1-8B truth probes

Walker, Humayun, Balestriero & Baraniuk (2026) propose the Linear Centroids Hypothesis and test it by re-running standard representation- analysis tools on centroids (mu = J^T*1, see [[jacobian-centroid-extraction]]) instead of raw activations, across four real, pretrained-model settings. (1) PCA on ResNet50's (ImageNet-pretrained) fourth-layer centroids separates two DTD texture classes into visually distinct linear directions; a companion FashionMNIST color-correlation probe shows centroid-space linear probes track a color feature only in proportion to its actual task-relevance, while activation-space probes stay linearly decodable for the feature even when it is spurious to the task. (2) TopK sparse dictionaries fit on DINOv2 and DINOv3 centroids show substantially higher cross-model max cosine similarity than dictionaries fit on the same models' raw activations, whose feature pairs cluster in a bimodal distribution around cosine similarity ~0.4 (read by the authors as poor cross-model correspondence) -- framed as additional support for [[platonic-representation]], since centroids are argued to track the functional mapping rather than incidental intermediate geometry. (3) On GPT2-Large, a centroid-displacement attribution score under single-neuron ablation independently re-identifies a previously reported "an"-predicting neuron (Clement & Joseph) at the 99.8th percentile of scores -- a representation-level circuit-discovery signal computed without reference to logits or output probabilities. (4) Mass-mean probes (Marks & Tegmark's construction) fit on Llama-3.1-8B centroids generalize better to held-out truth-identifying datasets than the same probes fit on raw activations. The paper also proves (Lemma 3.1, via a ReLU-collapse counterexample) that linear separability in raw activation space is necessary but not sufficient for two input sets to correspond to a genuine (non-spurious) feature -- motivating the shift to a Jacobian-anchored notion of feature in place of a purely geometric separability criterion. Several of the paper's own comparative numbers (exact probe accuracies/SDs, SAE firing-frequency rates, and the precise centroid-side cosine-similarity values) are reported only in figures that could not be independently extracted for this entry; the cosine-similarity ~0.4 baseline and the 99.8th-percentile neuron-attribution figure above are the two numerically quotable results recovered from the paper's running text.