MATH · IN · MODELS
structures / Manifolds / Cumulant-expansion profile of the next-token softmax entropy

Cumulant-expansion profile of the next-token softmax entropy

PROPERTYmeasurementfunctionaladvancedhow it's classified →

A computed profile of the higher-order cumulants (beyond mean and variance -- skewness, kurtosis, and further terms) of a language model's next-token softmax-entropy distribution, expanded as a series and tracked across layers/depth, characterizing the statistical shape of the predictive distribution's geometry rather than the shape of the activation point-cloud itself.

Replicationcomputed from the corpus — never hand-assigned
1 paper1 architecture class1 domain2 model families · one study
Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

For a language model’s next-token predictive distribution at a given layer/position, expand the softmax entropy (or an associated information-theoretic functional of the logit distribution) as a cumulant series: the mean (1st cumulant), variance (2nd), skewness- related 3rd cumulant, kurtosis-related 4th cumulant, and so on. Tracking how these cumulants evolve across layers or model depth yields a statistical-shape profile of the next-token-prediction (NTP) geometry — a different kind of “geometry” than the point-cloud shape of hidden-state activations: it characterizes the shape of the output probability distribution itself, induced by a representation, via its moment structure, rather than the spatial arrangement of representation vectors.

Relative to intrinsic-dimension-profile and manifold-curvature-profile

Intrinsic-dimension profile across depth and Curvature profile of the representation manifold both measure properties of the activation point-cloud — how many coordinates it needs, how bent it is. This node instead measures a property of the induced output distribution over the vocabulary at each layer/position — its higher-order statistical moments via a cumulant expansion — which is a functional (not a spatial-embedding) notion of geometry. A model could have very different point-cloud intrinsic dimension or curvature profiles while sharing similar cumulant-entropy profiles, or vice versa; the two families of measurement are orthogonal.

Key evidence

Viswanathan & Park (2025), “Probing Geometry of Next-Token Prediction Using Cumulant Expansion of the Softmax Entropy,” expand the mean softmax entropy of a prompt as a cumulant series around the “center” next-token distribution, using Tuned Lens logits at each layer of GPT-2 (small, large, XL) on Pile-10K prompts. Higher-order cumulants rise and plateau across layers for structured prompts and stay near zero for token-shuffled ones; in Pythia-160M they increase over training before saturating; and math prompts differ from general web text in middle-layer higher cumulants. See viswanathan-park-2025-cumulant-expansion-ntp-geometry.

Key papers

  • Probing Geometry of Next-Token Prediction Using Cumulant Expansion of the Softmax Entropy. arXiv:2510.04285 — origin of the cumulant- expansion NTP-geometry measurement.

Found in (1 observation · 2 families)

GPT-2

Probing Geometry of Next-Token Prediction Using Cumulant Expansion of the Softmax Entropy (2025)measured

Cumulant expansion of next-token softmax entropy tracks NTP distribution shape across layers

Details

- Using Tuned Lens to obtain per-layer next-token logits, the authors expand the mean softmax entropy of a prompt as a cumulant series around the "center" distribution (Eqs. 2-4) and track the cumulants layer by layer on Pile-10K prompts in GPT-2 (small, large, XL; main figures GPT-2 Large). For a structured prompt the higher-order cumulants rise through intermediate layers, plateau and decline slightly near the final layers, whereas for the token-shuffled version they stay near zero (Fig. 2); the paper reports the same pattern on other prompts and GPT-2 sizes (App. B). [viswanathan-park-2025-cumulant-expansion-ntp-geometry] - In Pythia-160M tracked across training, all cumulants increase before plateauing, and the gap between the entropy of the center distribution and the mean softmax entropy grows (Fig. 3). [viswanathan-park-2025-cumulant-expansion-ntp-geometry] - In GPT-2 Large, DM Mathematics prompts (99) versus Pile-CC prompts (570) show lower entropy, lower centre entropy, kappa_2 and kappa_3, but higher kappa_6 and kappa_7 in middle layers; the authors note the separability between the two distributions is modest (Fig. 4). The measurements are observational; the authors list causal manipulation of cumulants as future work. [viswanathan-park-2025-cumulant-expansion-ntp-geometry]

models: GPT-2 Small, GPT-2 Large, GPT-2 XL · method: Tuned Lens, Analytical derivation

Pythia

Probing Geometry of Next-Token Prediction Using Cumulant Expansion of the Softmax Entropy (2025)measured

Cumulant expansion of next-token softmax entropy tracks NTP distribution shape across layers

Details

- Using Tuned Lens to obtain per-layer next-token logits, the authors expand the mean softmax entropy of a prompt as a cumulant series around the "center" distribution (Eqs. 2-4) and track the cumulants layer by layer on Pile-10K prompts in GPT-2 (small, large, XL; main figures GPT-2 Large). For a structured prompt the higher-order cumulants rise through intermediate layers, plateau and decline slightly near the final layers, whereas for the token-shuffled version they stay near zero (Fig. 2); the paper reports the same pattern on other prompts and GPT-2 sizes (App. B). [viswanathan-park-2025-cumulant-expansion-ntp-geometry] - In Pythia-160M tracked across training, all cumulants increase before plateauing, and the gap between the entropy of the center distribution and the mean softmax entropy grows (Fig. 3). [viswanathan-park-2025-cumulant-expansion-ntp-geometry] - In GPT-2 Large, DM Mathematics prompts (99) versus Pile-CC prompts (570) show lower entropy, lower centre entropy, kappa_2 and kappa_3, but higher kappa_6 and kappa_7 in middle layers; the authors note the separability between the two distributions is modest (Fig. 4). The measurements are observational; the authors list causal manipulation of cumulants as future work. [viswanathan-park-2025-cumulant-expansion-ntp-geometry]

models: Pythia-160M · method: Tuned Lens, Analytical derivation