Non-orthogonal steering directions mutually dampen; GEMS restores them

measured in 1 paper

Deng (GEMS) shows three diff-in-means persona directions (empathy, accountability, minimalism) in Qwen3.5-4B are pairwise cosine-similar at 0.74-0.87 [deng-2026-gems-geometric-constraints-multi-semantic-superposition-llms] Injecting all three simultaneously degrades each direction's terminal-layer alignment from 0.64-0.66 to 0.38-0.43, a 34-43% mutual reduction attributed to non-orthogonality [deng-2026-gems-geometric-constraints-multi-semantic-superposition-llms] Unconstrained additive injection separately drives residual-stream norm to 3.4x baseline, flattening next-token probability (top-1 43% to 11%) [deng-2026-gems-geometric-constraints-multi-semantic-superposition-llms] GEMS applies per-token Gram-Schmidt orthogonalization plus norm-constrained recombination restricted to the attention-output pathway [deng-2026-gems-geometric-constraints-multi-semantic-superposition-llms] On GSM8K with 3 concurrent directions, naive steering collapses accuracy 92% to 4% while GEMS recovers 98% at only +2.2% perplexity [deng-2026-gems-geometric-constraints-multi-semantic-superposition-llms]

Structure

Linear Direction

Context

pairwise cosine similarity between steering directions as a direct, quantified predictor of mutual dampening under naive superposition, out-of-distribution residual-stream norm accumulation as a distinct, independently-measured collapse mode, Gram-Schmidt orthogonalization plus norm-constrained recombination as a geometry-tied fix for multi-direction steering

Confirmed in models

Qwen3.5-4B · Qwen, Qwen3.5-4B-Instruct · Qwen, Llama-3.2-3B-Instruct · Llama, Qwen3.6-27B-Instruct · Qwen, Gemma-4-31B-Instruct · Gemma

Method

Difference-in-means direction, Orthogonalized Multi-Direction Superposition Steering (GEMS)

Papers

GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs — Deng, Yu2026 · arXiv:2606.19946