Definition
Given a labeled concept class (e.g. “harmful instructions,” “toxic text”), cluster its activations with an unsupervised method (k-means, Gaussian Mixture Model) into groups. Concept cluster heterogeneity is confirmed when clusters are needed to fit the data well (vs. a single Gaussian), and post-hoc inspection shows each cluster corresponds to a coherent semantic subgroup rather than an arbitrary partition — meaning the class is a mixture of distinguishable sub-populations, not a single tight blob around one centroid.
Intuition
A single steering/editing vector computed as one global difference-in-means implicitly assumes a concept is a single point (its class mean) that can be reached by one fixed translation. If “harmful instructions” is really several distinguishable sub-types (violence, fraud, self-harm, etc.) occupying different regions of activation space, a single mean-difference vector is a poor summary of any of them individually — it is instead the centroid of a mixture, which may lie in a low-density region between the true sub-clusters.
Properties
- Discovered, not assumed. The clustering is fit on real model activations and validated (e.g. cluster count selected via information criteria, or clusters shown to align with human-labeled subgroups on manual inspection) rather than a modeling choice imposed a priori.
- Degenerate case recovers the single-direction picture. When best fits the data, the concept is unimodal and a single Difference-in-means direction direction is an adequate summary — heterogeneity is a generalization of, not a replacement for, the single-direction case.
- Low-rank steering-covariance structure. Empirically, the covariance of the per-cluster steering directions needed to reach a heterogeneous target concept can itself be low-rank (e.g. concentrated in the top principal components for clusters), meaning the heterogeneity, while real, is still a compact structure rather than requiring full-rank freedom.
- Causal payoff. Exploiting the discovered cluster structure (e.g. via a per-cluster optimal-transport steering map, see GMM-Optimal-Transport steering (CHaRS)) rather than a single global mean-difference vector can produce a measurably stronger causal steering effect than the single-direction baseline.
Key evidence
Abdullaev, Wong, Lee et al. (2026) show PCA/t-SNE visualizations of last-token hidden states in Llama-3.2-3B-Instruct and Qwen2.5-Instruct models revealing clustered, multimodal structure within nominally single concept classes, with manual inspection confirming clusters correspond to coherent semantic subgroups (e.g. distinct categories of harmful instruction). They further show the steering-direction covariance implied by their per-cluster optimal-transport map is concentrated in the top principal components, matching a derived theoretical rank bound, and that exploiting this heterogeneity (via GMM-Optimal-Transport steering (CHaRS)) raises jailbreak attack-success rate and improves toxicity-mitigation trade-offs relative to single-direction (diff-in-means/ActAdd) steering baselines.
Anderson (2026) applies K-means + silhouette-score mode discovery (PCA
dimensionality reduction, k-means sweep, maximum silhouette score) to
chain-of-thought reasoning trajectories in real Llama-3-8B-Instruct and
Llama-3.1-70B-Instruct on code-reasoning tasks, finding the population of
trajectories increasingly resolves into a small number of discrete,
well-separated clusters (“strategic modes”) with scale — maximum
silhouette rising 0.13->0.42 from the 8B to the 70B model — the paper’s
own framing being a discrete “Lattice” of modes, contrasted with legal
reasoning’s smooth dimensional-collapse pattern at the same two scales
(see Dimensional collapse). Here the “nominal concept class” being
found heterogeneous is a single reasoning-trajectory population within
one task domain, rather than a labeled semantic concept class. See
anderson-2026-code-reasoning-trajectories-in-real-llama-3-8b-instruct-and-llama-3-1-70b-instruct-organize-into-a-small-number-of-discrete-silhouette-validated-clusters-that-sharpen-with-scale.
A real trained contrastive vision MoE-CNN’s experts cluster into a stable animate/inanimate partition via RSA, across expert counts and independently trained seeds. Tangtartharakul & Storrs (2026) train a custom sparsely-gated contrastive MoE-CNN (ResNet backbone, SimCLR, E in {4,8,16} experts) from scratch on STL10, and find agglomerative clustering of per-expert representational-similarity matrices (built on the independent THINGS dataset) recovers a robust 2-cluster animate/inanimate partition — here the “nominal concept class” being unexpectedly heterogeneous is not a single labeled behavior category but an entire population of expert representations, which nonetheless resolves into exactly the same kind of discovered, RSA-validated sub-cluster structure — stable across expert counts and 10 independently trained seeds, with complementary Lasso regression showing individual experts are broadly tuned to continuous dimensions beyond hard category boundaries. See tangtartharakul-storrs-2026-a-real-trained-contrastive-vision-moe-cnns-experts-cluster-into-a-stable-animate-inanimate-partition-via-rsa-across-expert-counts-and-independently-trained-seeds.