MATH · IN · MODELS
methods / Theoretical / Analytical / Geometric analysis / Cross-expert Jacobian alignment

Cross-expert Jacobian alignment

Techniqueadvanced

Measures functional (not merely representational) similarity between two experts in a mixture-of-experts layer by comparing the cosine similarity of their local input-output Jacobians at shared inputs, rather than comparing their weights or output activations directly — a near-zero value indicates the experts implement decorrelated functions even if their routed representations occupy overlapping subspaces.

Used in (1 observation)

structure: Linear Subspace · models: Mixtral-8x7B, Qwen1.5-MoE-A2.7B · paper: Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap