Weight decay collapses ordinal embeddings onto a monotone end-anchored line

Singh proves (Proposition 1, a minimum-norm/weight-decay argument) that a transformer trained only on adjacent-pair comparisons over a total order collapses its embeddings onto a collinear, monotonically rank-ordered line with end-anchor stretching [singh-2026-emergent-ordinal-geometry] A trained single-layer toy transformer confirms this, its peak-checkpoint embeddings forming a monotone curve (rank correlation 0.96) with the predicted symbolic-distance and end-anchor effects [singh-2026-emergent-ordinal-geometry] Probing pretrained Qwen2.5-1.5B recovers the same monotone line across three ordinal domains with domain-dependent fidelity: digits cleanest (r~0.81 on PC2), sizes moderate, and months weakest (r=0.41) because a calendar is cyclic and resists the line [singh-2026-emergent-ordinal-geometry] Per-domain probe directions are near-orthogonal (|cos|<0.2), so distinct ordinal concepts occupy separate directions of the same shape [singh-2026-emergent-ordinal-geometry]