Cumulant expansion of next-token softmax entropy tracks NTP distribution shape across layers

- Using Tuned Lens to obtain per-layer next-token logits, the authors expand the mean softmax entropy of a prompt as a cumulant series around the "center" distribution (Eqs. 2-4) and track the cumulants layer by layer on Pile-10K prompts in GPT-2 (small, large, XL; main figures GPT-2 Large). For a structured prompt the higher-order cumulants rise through intermediate layers, plateau and decline slightly near the final layers, whereas for the token-shuffled version they stay near zero (Fig. 2); the paper reports the same pattern on other prompts and GPT-2 sizes (App. B). [viswanathan-park-2025-cumulant-expansion-ntp-geometry] - In Pythia-160M tracked across training, all cumulants increase before plateauing, and the gap between the entropy of the center distribution and the mean softmax entropy grows (Fig. 3). [viswanathan-park-2025-cumulant-expansion-ntp-geometry] - In GPT-2 Large, DM Mathematics prompts (99) versus Pile-CC prompts (570) show lower entropy, lower centre entropy, kappa_2 and kappa_3, but higher kappa_6 and kappa_7 in middle layers; the authors note the separability between the two distributions is modest (Fig. 4). The measurements are observational; the authors list causal manipulation of cumulants as future work. [viswanathan-park-2025-cumulant-expansion-ntp-geometry]