MATH · IN · MODELS
methods / Theoretical / Analytical / Geometric analysis / Router-expert weight coupling analysis

Router-expert weight coupling analysis

Techniqueadvanced

Measures cosine similarity among a sparse mixture-of-experts model's learned router weight vectors, and correlates router scores with real per-token expert-gate activations, to test whether router and expert weights become geometrically aligned because they receive gradient updates proportional to the same routed hidden-state direction.

Used in (1 observation)

structure: Linear Direction · models: Custom 1B-parameter sparse MoE transformer (Ahrac et al. 2026) · paper: Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts