Separability/alignment decomposition

methods / Theoretical / Analytical / Separability/alignment decomposition

Techniqueintermediate

Proves classification accuracy is upper-bounded by the maximum linear separability of hidden states, with equality when the separating direction also aligns with the label unembedding-difference direction — then measures both quantities layer-by-layer to explain when and why interventions work.

Used in (1 observation)

ICL separability rises first, then alignment with the unembeddingmeasured

structure: Linear Direction, Linear Separability · models: Llama-2-7B, Llama-2-13B, Llama-2-70B, Llama-3-8B, Llama-3-70B, Gemma-2B, Gemma-7B · paper: Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning