Evaluation-vs-deployment diff-in-means direction in six open LLMs: decodable, weakly coupled to verbalization, steerable

A per-layer diff-in-means direction from 16 paired deployment/evaluation prompts separates WildChat from GSM8K/TruthfulQA/MASK prompts with best-layer AUROC above 0.7 in Qwen3-8B, Qwen3-32B, Olmo3-7B, Olmo3-32B, Gemma4-31B and Llama3.3-Nemotron-49B [heidari-etal-2026-evaluation-awareness] Per-prompt probe score vs. LLM-judge verbalization score has |Spearman rho| below 0.19 and mutual information below 0.04 nats across all models and layers, close to a random-direction baseline [heidari-etal-2026-evaluation-awareness] Adding the direction (Contrastive-Activation-Addition style, norm-matched) at negative strength shifts the mean judge score by about 0.2-1.5 points on a 0-3 scale in Olmo3-7B/32B Think and Qwen3-8B; Qwen3-32B is irregular with degenerate outputs [heidari-etal-2026-evaluation-awareness] Across Olmo3 Base/SFT/DPO/Think the SFT, DPO and Think directions have cosine at least 0.998, while the Base direction differs (cosine 0.72 at 7B, 0.83 at 32B) [heidari-etal-2026-evaluation-awareness] Evidence is a single linear direction per layer on one prompt operationalization; the authors list nonlinear and multi-direction interventions as untested [heidari-etal-2026-evaluation-awareness]