RSA-localized concept vectors are surface-form invariant and causally guide behavior

Opielka et al. use RSA to localize the top-3 attention heads in Llama-3.1-8B and Llama-3.1-70B whose summed mean activations form concept vectors [opielka-etal-2025-analogical-concept-vectors] For verbal concepts (antonym, category) the concept vectors are invariant across surface form (cross-context cosine ~0.7-0.8), while abstract relational concepts (previous, next) find no comparable invariant [opielka-etal-2025-analogical-concept-vectors] Adding a concept vector into the residual stream causally steers behavior toward the target concept and transfers out-of-distribution better than function vectors [opielka-etal-2025-analogical-concept-vectors] Concept vectors are markedly weaker at pure zero-shot intervention and need in-context support, and activation patching is shown to be an unreliable localization tool for these components [opielka-etal-2025-analogical-concept-vectors]