MATH · IN · MODELS
methods / Theoretical / Analytical / Geometric analysis / Activation-cluster functional typing

Activation-cluster functional typing

Techniqueintermediate

Partitions a population of tokens into discrete functional categories by clustering their activation vectors (via a norm threshold plus cross-token cosine similarity of the resulting cluster centroids), then validates each category's functional distinctness via downstream ablation — used to show that a nominally homogeneous set of tokens (e.g. all visual tokens fed into a multimodal LLM) actually splits into a small number of geometrically near-collapsed sub-populations with sharply different causal roles.

Used in (1 observation)

structure: Anisotropy · models: LLaVA-1.5-7B, LLaVA-1.5-13B, InternVL3-8B · paper: What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models