RoPE massive values concentrate at fixed Q/K dimensions
measured in 1 paper- In RoPE transformers, "massive values" (per-head L2 norm >5x the mean) concentrate at consistent head-dimension indices in query/key space after RoPE, and are absent in non-RoPE models. [jin-etal-2025-massive-values-self-attention] - Disrupting the massive-value dimensions at prefill collapses contextual-knowledge tasks (Llama-3-8B GSM8K 76.9 to 4.0, IMDB 95.4 to 11.0, Passkey 100 to 9.0) while parametric knowledge survives (Cities 95.0 to 88.2); disrupting non-massive dimensions changes accuracy <+-1%. [jin-etal-2025-massive-values-self-attention] - Quantization that protects large-magnitude channels (AWQ, SmoothQuant) preserves contextual accuracy, whereas GPTQ, which does not, degrades it. [jin-etal-2025-massive-values-self-attention] - Main results on Llama-3-8B-Instruct, Gemma-2-9B-it and Qwen2.5-7B-Instruct, with Llama-2-7B-chat and non-RoPE GPT-2/OPT as contrasts (further appendix models not re-verified here). [jin-etal-2025-massive-values-self-attention]