Feature Lobes (Spatial-Functional Modularity)

structures / Manifolds / Feature Lobes (Spatial-Functional Modularity)

Feature Lobes (Spatial-Functional Modularity)

PROPERTYempirical-patternintermediatehow it's classified →

A partition of a large feature point cloud into spatially-coherent regions ('lobes'), discovered purely from functional co-occurrence statistics with no positional information used, that nonetheless turn out to occupy distinguishable spatial regions — an intermediate-scale organizing principle between single-direction/subspace structure and the point cloud's global shape.

Replicationcomputed from the corpus — never hand-assigned

1 paper1 architecture class1 domain1 model family

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

Given a large set of feature vectors $\{f_i\}\subset\mathbb{R}^d$ (e.g. an SAE dictionary’s decoder directions) and a functional relatedness signal computed without using each feature’s position (e.g. how often two features co-fire within the same document), partition the features into $k$ groups (“lobes”) that maximize within-group functional relatedness. A feature lobe structure is confirmed when this purely-functional partition, evaluated after the fact against the features’ actual positions in $\mathbb{R}^d$ , shows significantly more spatial coherence (features assigned to the same lobe cluster together in space) than a null model in which functional relatedness and spatial position are independent.

Intuition

Named by analogy to functional lobes in fMRI brain imaging (a loose naming metaphor, not a literal anatomical claim): groups of biological neurons that tend to activate together for related tasks (language, vision, motor control) also happen to be spatially clustered in the brain, rather than being scattered evenly and merely “wired together” at a distance. This structure asks the analogous question of an artificial network’s feature space — do functionally-related features (e.g. all the ones that fire on math or code text) also occupy a shared region of representation space, or does functional relatedness carry no spatial signature at all?

Properties

Discovered functionally, confirmed spatially — never the reverse. The defining methodology is strict about direction: lobes are identified from co-occurrence/functional statistics alone, with spatial position withheld during discovery, and only checked against position afterward — this order rules out the trivial confound of a spatial clustering method simply re-deriving its own input.
A coarser scale than any single Linear Direction or Linear Subspace. A lobe is not a claim about the geometry of any one feature or concept; it is a claim about how many features’ positions relate to many features’ functional relationships in aggregate — the intermediate scale between a single feature’s shape (Concept Crystals (Parallelogram/Trapezoid Structure), atomic scale) and the global shape of the entire point cloud (Intrinsic-dimension profile across depth-style depth-wise structure, galaxy scale).
Requires an explicit null model, not just “more clustered than uniform.” Spatial coherence must be compared against a stated null hypothesis (e.g. random permutation of lobe labels across positions, or spatial statistics computed on randomly-initialized decoder directions) — a high absolute clustering score is not by itself evidence of a real functional-spatial correspondence unless it’s shown to exceed what the same statistic would produce by chance.
Coarseness-dependent. The same feature population can show significant functional-spatial correspondence at a coarse partition (e.g. 2-3 broad lobes: code/math vs. natural language) while finer-grained sub-partitions may show progressively weaker or noisier correspondence — the claim is scale-dependent, not a single universal statement true at every granularity simultaneously.
A structural/organizational claim, not a causal one on its own. Finding that functionally-related features cluster spatially says the network’s representation has a certain macroscopic organization; it does not by itself establish that this spatial arrangement is causally exploited by any downstream computation (e.g. via local receptive fields or spatially-restricted routing) — that would require a separate causal test.

Exercises

Base

Why is it important that lobes are discovered using only functional co-occurrence data, with spatial position withheld, rather than discovered directly from spatial clustering of the feature vectors?

Solution

If lobes were discovered by spatially clustering the feature vectors directly, then checking whether spatially-clustered groups are spatially coherent would be circular — of course a group defined by spatial proximity is spatially coherent. Discovering the groups from an independent signal (functional co-occurrence) and only afterward checking spatial coherence is what makes “functionally-related features are also spatially clustered” a genuine, falsifiable empirical claim rather than a tautology.

A researcher reports “our lobes have high internal spatial density” without comparing to any null model. What is missing from this claim, and why does it matter?

Solution

Missing is a comparison baseline: “high” internal spatial density is meaningless without knowing what density would be expected under a null hypothesis of no functional-spatial correspondence (e.g. random groupings of the same sizes, or co-occurrence-based groupings applied to randomly-initialized feature directions). Without this comparison, an apparently high density could simply reflect the fact that any random subset of points in a bounded, non-uniformly-populated space will show some apparent local density — the claim only becomes evidence for the structure once it’s shown to exceed a stated chance baseline, ideally by a quantified, statistically significant margin.

Middle

Suppose two independent functional-relatedness metrics (e.g. one based on strict co-occurrence counts, another based on cosine similarity of feature vectors’ own directions) are both used to partition the same feature population, and the two partitions agree strongly (high mutual information). Does this by itself establish the feature-lobes structure? What additional check is still needed?

Solution

High agreement between two functional-relatedness metrics establishes that the functional signal is robust/reproducible across different ways of measuring functional relatedness — but it says nothing about whether that shared functional grouping corresponds to anything spatial. The feature-lobes claim specifically requires cross-referencing a functionally-derived partition against the features’ actual spatial positions, so even a highly self-consistent functional partition still needs the separate step of checking it against position (and a null model for that spatial check) before the lobes structure itself is established — robustness of the functional signal and existence of a spatial correspondence are two different claims that both need independent support.

A 2-lobe partition shows strong, highly-significant spatial coherence, but a 10-lobe partition of the same data shows much weaker (though still statistically significant) coherence. Is this necessarily a contradiction, or a sign the method failed at fine granularity? Explain.

Solution

Not necessarily a contradiction or a failure — per the coarseness-dependence property, functional-spatial correspondence can genuinely be strongest at a coarse scale (broad domains like code/math vs. natural language occupying clearly separated regions) while becoming progressively noisier at finer granularity (individual sub-topics within “natural language” may be functionally distinguishable but not occupy cleanly separated sub-regions of the same broad lobe) — this is consistent with a real, but scale-dependent, organizing principle, rather than evidence the whole finding is spurious, provided the finer-grained result still clears its own (separately calibrated) significance threshold.

Pro

Propose a specific statistical test (analogous to the significance tests used for categorical-polytope or hierarchical-orthogonality elsewhere on this map) that would let you report a precise “how many standard deviations from chance” figure for a claimed feature-lobes structure, using a permutation-based null model.

Solution

Compute the chosen spatial-coherence statistic (e.g. mean intra-lobe pairwise distance, or a clustering-quality score comparing the functionally-derived lobe labels against a position-based clustering) on the real data, call this $T_{\text{obs}}$ . Then generate a large number $N$ of permuted null datasets by randomly reassigning the functionally-derived lobe labels to feature positions (breaking any real functional-spatial association while preserving the label distribution and the spatial point cloud’s own structure), computing the same statistic $T_i$ for each permutation. The null distribution $\{T_1,\ldots,T_N\}$ gives an empirical mean $\bar T$ and standard deviation $\sigma_T$ ; report $z = (T_{\text{obs}}-\bar T)/\sigma_T$ as the number of standard deviations the observed spatial coherence exceeds chance — a standard permutation-test z-score, directly analogous to significance tests used elsewhere on this map for other structural claims, and applicable regardless of the specific spatial-coherence statistic chosen, since it only requires that the statistic be computable identically on both real and permuted label assignments.

Found in (1 observation · 1 family)

Gemma

The Geometry of Concepts: Sparse Autoencoder Feature Structure (2024)measured

Co-occurring SAE features cluster into spatial functional lobes

Details

- Using the Gemma-2-2B Gemma Scope SAE (layer 12, 16k features, average L0=41), functional co-occurrence of features (co-firing within 256-token blocks over 50,000 Pile documents) is spectrally clustered via five affinity measures, without using decoder-vector positions. [li-etal-2024-geometry] - The purely functional partition matches the features' actual spatial positions: a 2-lobe split separates code/math from natural-language features, and a 3-lobe split further separates short-message/dialogue from long-form scientific text. [li-etal-2024-geometry] - The phi coefficient gives the strongest functional-spatial correspondence, significant at 954 standard deviations (vs a permuted-label null) and 74 (vs a random-decoder null). [li-etal-2024-geometry] - "Lobes" is a naming metaphor to functional brain lobes (functionally related plus spatially clustered), not a claim of anatomical correspondence. [li-etal-2024-geometry]

models: Gemma-2-2B · method: Sparse Autoencoders (SAE), Co-occurrence affinity clustering