MATH · IN · MODELS
methods / Direction Extraction / Concept-Token Attention-Output Projection (ConceptAttention)

Concept-Token Attention-Output Projection (ConceptAttention)

Techniqueintermediate

Thread extra 'concept' tokens through a frozen diffusion transformer's own text-conditioning weights, one-directionally (concepts read from image/prompt tokens, never write back), then localize each concept by a dot product between concept and image tokens taken specifically in the attention-output space (post output-projection) rather than the raw cross-attention softmax(QK^T)V space.

Used in (2 observations)

structure: Linear Direction · models: FLUX.1 [schnell] · paper: ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features
structure: Linear Direction · models: Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct · paper: Steering at the Source: Style Modulation Heads for Robust Persona Control