Concept directions anti-concentrate in the unembedding's low-variance spectral tail

measured in 1 paper

Acharya, Rimal & Dhakal project concept diff-of-means vectors onto the unembedding-covariance eigenbasis across 17 models in 5 families [acharya-rimal-dhakal-2026-spectral-anti-concentration] Concept-direction energy anti-concentrates in the low-eigenvalue tail: Spectral Center of Mass averages 0.926 versus 0.758 for random directions (Gini-deviation -0.282) [acharya-rimal-dhakal-2026-spectral-anti-concentration] Static unembedding-row contrasts show the opposite sign, and POS/syntax directions concentrate in the high-variance subspace in 6 of 8 architectures (p<0.013), a genuine dual-geometry split [acharya-rimal-dhakal-2026-spectral-anti-concentration] On Llama-3.1-8B, injecting only the top-10%-eigenvalue component gives +383% perplexity versus +223% for the bottom-10% component (Cohen's d=1.80), replicated across 5 models [acharya-rimal-dhakal-2026-spectral-anti-concentration]

Structure

Linear Direction

Context

spectral center of mass (SCM), Gini-deviation spectral anti-concentration score, unembedding covariance eigenbasis, dual geometry (concept vs. syntax spectral location), split-injection steering (top-10% vs. bottom-10% eigencomponents)

Confirmed in models

Llama-3.2-1B · Llama, Llama-3.2-3B · Llama, Llama-3.1-8B · Llama, Qwen2.5-0.5B · Qwen, Qwen2.5-1.5B · Qwen, Qwen2.5-3B · Qwen, Qwen-2.5-7B · Qwen, Qwen2.5-14B · Qwen, Qwen3-0.6B · Qwen, Qwen3-1.7B · Qwen, Qwen3-4B · Qwen, Qwen3-8B · Qwen, Gemma-2-2B · Gemma, Gemma-2-9B · Gemma, Mistral-7B-v0.3 · Mistral, SmolLM2-1.7B · SmolLM2, OLMoE-1B-7B · OLMoE, JetMoE-8B · JetMoE

Method

Geometric analysis, Causal interventions (steering)

Papers

Concepts Whisper While Syntax Shouts: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations — Acharya, Pratyush, Rimal, Nuraj, Dhakal, Habish2026 · arXiv:2605.01609