MATH · IN · MODELS
structures / Linear Structures / Attention-graph spectral profile (Fiedler value, HFER, smoothness, spectral entropy)

Attention-graph spectral profile (Fiedler value, HFER, smoothness, spectral entropy)

PROPERTYmeasurementfunctionaladvancedhow it's classified →

Treating a transformer's attention matrix at each layer as the weighted adjacency matrix of a dynamic graph over tokens, and reducing it to four scalar diagnostics from its graph Laplacian spectrum: the Fiedler value (algebraic connectivity), high-frequency energy ratio (HFER), graph-signal smoothness (Dirichlet energy of hidden states as a graph signal), and spectral entropy of the Laplacian eigenvalue distribution.

Replicationcomputed from the corpus — never hand-assigned
1 paper1 architecture class1 domain4 model families · one study
Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

Given an attention matrix A∈Rn×nA\in\mathbb{R}^{n\times n} at a layer (after some head-aggregation rule combining multiple heads into one weighted graph over the nn tokens), form the graph Laplacian L=D−AL=D-A (or a normalized variant), with eigenvalues 0=λ1≤λ2≤⋯≤λn0=\lambda_1\le\lambda_2\le\cdots\le\lambda_n. Four scalar diagnostics reduce this per-layer spectrum to a compact profile:

  • Fiedler value / algebraic connectivity λ2\lambda_2 — how well- connected the token graph is (low λ2\lambda_2 = graph is close to splitting into loosely-connected clusters; high λ2\lambda_2 = uniformly well-mixed).
  • High-frequency energy ratio (HFER) — treating a hidden-state feature as a graph signal xx, the fraction of xx‘s energy that projects onto high-eigenvalue (rough, rapidly-varying-across-edges) Laplacian eigenvectors, versus low-eigenvalue (smooth) ones.
  • Graph-signal smoothness — the Dirichlet-energy quadratic form x⊤Lx=∑(i,j)Aij(xi−xj)2x^\top L x = \sum_{(i,j)} A_{ij}(x_i-x_j)^2, small when connected tokens carry similar feature values.
  • Spectral entropy — the Shannon entropy of the normalized Laplacian eigenvalue distribution {λi/∑jλj}\{\lambda_i/\sum_j\lambda_j\}, low when a few eigenvalues dominate (a simple, low-complexity graph structure), high when eigenvalues are spread evenly.

Relative to attention-reference-frame

Attention reference frame (sink-token anchor configuration) also treats attention as a graph and uses the Fiedler value, but as one diagnostic among several used to distinguish reference-frame configuration types (centralized vs. distributed vs. bidirectional token anchoring). This node instead treats the full four-diagnostic spectral profile as a general-purpose, task-agnostic measurement — a per-input, per-layer fingerprint that can be correlated with any downstream property (here: mathematical-proof validity), not tied to any one anchor-point configuration.

Key evidence

Noël (2026) computes this four-diagnostic profile per layer across seven models from four architecturally-independent families (Llama-3.2-1B/3B, Llama-3.1-8B, Qwen2.5-0.5B/7B, Phi-3.5-mini, Mistral-7B-v0.1) on Lean/Isabelle-style formal mathematical proofs, finding statistically significant differences between spectrally-valid and spectrally-invalid proofs in every model (effect sizes up to Cohen’s d=3.30d=3.30, p<10−116p<10^{-116}), enabling training-free classification of proof validity at 85.0-95.6% accuracy from a single calibrated threshold on one spectral metric (no learned classifier, fine-tuning, or training data required). Global- attention models (Llama, Qwen, Phi) show their strongest discriminative signal in HFER; Mistral-7B, the one model using Sliding Window Attention rather than global attention, shows the signal shift to late-layer smoothness instead (d=2.09d=2.09, Mann-Whitney p=1.16×10−48p=1.16\times10^{-48}) — evidence that attention-mechanism design determines which spectral diagnostic carries the reasoning-validity signal, not merely its magnitude. Robustness controls (model-generated valid-vs-invalid pairs; human-authored logic-corrupting perturbations holding writing style fixed) and a label-correction analysis both support that the spectral signature tracks genuine logical coherence rather than compiler/verifier acceptance — some proofs the spectral method flags as valid are rejected by formal verifiers only for technical reasons (timeouts, missing imports), not logical error. See attention-graph-spectral-diagnostics-distinguish-valid-from-invalid-mathematical-proofs-across-seven-models-and-four-architecture-families.

How to detect it

Aggregate multi-head attention into a single weighted graph per layer (e.g. average or max over heads), compute the graph Laplacian and its eigendecomposition, then extract the Fiedler value, HFER (energy split at some eigenvalue cutoff), Dirichlet-energy smoothness of a chosen hidden-state signal, and Laplacian spectral entropy. Compare the resulting per-layer profiles between two conditions of interest (e.g. valid vs. invalid reasoning) via effect size and significance testing, checking which diagnostic(s) discriminate and at which layers.

Key papers

  • Noël, V. (2026). Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning. arXiv:2601.00791 — origin of this four- diagnostic attention-graph spectral profile and its application to training-free mathematical-reasoning validity detection.

Found in (1 observation · 4 families)

Llama

Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning (2026)measured

Attention-graph Laplacian spectra separate valid from invalid proofs

Details

- Treating each layer's attention matrix as a weighted graph, four training-free Laplacian spectral diagnostics (Fiedler value lambda2, high-frequency energy ratio HFER, spectral entropy and smoothness) separate valid from invalid mathematical proofs. [noel-2026-geometry-of-reason] - The separation is large and highly significant across all seven models (Cohen's d 2.09-3.30; per-model Mann-Whitney p from 1.16e-48 to 4.51e-66; peak d=3.30 for Phi-3.5-mini, with a pooled significance p<1e-116). [noel-2026-geometry-of-reason] - A single calibrated threshold on one spectral metric classifies proofs at 85.0-95.6% accuracy (93-95% on the full dataset), dropping to 82.8-85.9% under nested cross-validation. [noel-2026-geometry-of-reason] - Global-attention models (Llama, Qwen, Phi) carry the signal in HFER, whereas Mistral-7B's Sliding-Window Attention shifts it to late-layer smoothness (d=2.09, p=1.16e-48). [noel-2026-geometry-of-reason] - Some proofs the spectral method flags as valid are rejected by Lean/Isabelle only for technical reasons (timeouts, missing imports), which the paper terms "Platonic validity". [noel-2026-geometry-of-reason] - Tested on MiniF2F proofs across seven models / four families (Llama-3.2-1B/3B, Llama-3.1-8B, Qwen2.5-0.5B/7B, Phi-3.5-mini, Mistral-7B-v0.1 — named as base checkpoints though the text calls them instruction-tuned); observational. [noel-2026-geometry-of-reason]

models: Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B · method: Geometric analysis

Qwen

Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning (2026)measured

Attention-graph Laplacian spectra separate valid from invalid proofs

Details

- Treating each layer's attention matrix as a weighted graph, four training-free Laplacian spectral diagnostics (Fiedler value lambda2, high-frequency energy ratio HFER, spectral entropy and smoothness) separate valid from invalid mathematical proofs. [noel-2026-geometry-of-reason] - The separation is large and highly significant across all seven models (Cohen's d 2.09-3.30; per-model Mann-Whitney p from 1.16e-48 to 4.51e-66; peak d=3.30 for Phi-3.5-mini, with a pooled significance p<1e-116). [noel-2026-geometry-of-reason] - A single calibrated threshold on one spectral metric classifies proofs at 85.0-95.6% accuracy (93-95% on the full dataset), dropping to 82.8-85.9% under nested cross-validation. [noel-2026-geometry-of-reason] - Global-attention models (Llama, Qwen, Phi) carry the signal in HFER, whereas Mistral-7B's Sliding-Window Attention shifts it to late-layer smoothness (d=2.09, p=1.16e-48). [noel-2026-geometry-of-reason] - Some proofs the spectral method flags as valid are rejected by Lean/Isabelle only for technical reasons (timeouts, missing imports), which the paper terms "Platonic validity". [noel-2026-geometry-of-reason] - Tested on MiniF2F proofs across seven models / four families (Llama-3.2-1B/3B, Llama-3.1-8B, Qwen2.5-0.5B/7B, Phi-3.5-mini, Mistral-7B-v0.1 — named as base checkpoints though the text calls them instruction-tuned); observational. [noel-2026-geometry-of-reason]

models: Qwen2.5-0.5B, Qwen-2.5-7B · method: Geometric analysis

Phi

Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning (2026)measured

Attention-graph Laplacian spectra separate valid from invalid proofs

Details

- Treating each layer's attention matrix as a weighted graph, four training-free Laplacian spectral diagnostics (Fiedler value lambda2, high-frequency energy ratio HFER, spectral entropy and smoothness) separate valid from invalid mathematical proofs. [noel-2026-geometry-of-reason] - The separation is large and highly significant across all seven models (Cohen's d 2.09-3.30; per-model Mann-Whitney p from 1.16e-48 to 4.51e-66; peak d=3.30 for Phi-3.5-mini, with a pooled significance p<1e-116). [noel-2026-geometry-of-reason] - A single calibrated threshold on one spectral metric classifies proofs at 85.0-95.6% accuracy (93-95% on the full dataset), dropping to 82.8-85.9% under nested cross-validation. [noel-2026-geometry-of-reason] - Global-attention models (Llama, Qwen, Phi) carry the signal in HFER, whereas Mistral-7B's Sliding-Window Attention shifts it to late-layer smoothness (d=2.09, p=1.16e-48). [noel-2026-geometry-of-reason] - Some proofs the spectral method flags as valid are rejected by Lean/Isabelle only for technical reasons (timeouts, missing imports), which the paper terms "Platonic validity". [noel-2026-geometry-of-reason] - Tested on MiniF2F proofs across seven models / four families (Llama-3.2-1B/3B, Llama-3.1-8B, Qwen2.5-0.5B/7B, Phi-3.5-mini, Mistral-7B-v0.1 — named as base checkpoints though the text calls them instruction-tuned); observational. [noel-2026-geometry-of-reason]

models: Phi-3.5-mini · method: Geometric analysis

Mistral

Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning (2026)measured

Attention-graph Laplacian spectra separate valid from invalid proofs

Details

- Treating each layer's attention matrix as a weighted graph, four training-free Laplacian spectral diagnostics (Fiedler value lambda2, high-frequency energy ratio HFER, spectral entropy and smoothness) separate valid from invalid mathematical proofs. [noel-2026-geometry-of-reason] - The separation is large and highly significant across all seven models (Cohen's d 2.09-3.30; per-model Mann-Whitney p from 1.16e-48 to 4.51e-66; peak d=3.30 for Phi-3.5-mini, with a pooled significance p<1e-116). [noel-2026-geometry-of-reason] - A single calibrated threshold on one spectral metric classifies proofs at 85.0-95.6% accuracy (93-95% on the full dataset), dropping to 82.8-85.9% under nested cross-validation. [noel-2026-geometry-of-reason] - Global-attention models (Llama, Qwen, Phi) carry the signal in HFER, whereas Mistral-7B's Sliding-Window Attention shifts it to late-layer smoothness (d=2.09, p=1.16e-48). [noel-2026-geometry-of-reason] - Some proofs the spectral method flags as valid are rejected by Lean/Isabelle only for technical reasons (timeouts, missing imports), which the paper terms "Platonic validity". [noel-2026-geometry-of-reason] - Tested on MiniF2F proofs across seven models / four families (Llama-3.2-1B/3B, Llama-3.1-8B, Qwen2.5-0.5B/7B, Phi-3.5-mini, Mistral-7B-v0.1 — named as base checkpoints though the text calls them instruction-tuned); observational. [noel-2026-geometry-of-reason]

models: Mistral-7B-v0.1 · method: Geometric analysis