MATH · IN · MODELS
methods / Theoretical / Analytical / Next-token-prediction SVD orthant decomposition

Next-token-prediction SVD orthant decomposition

Techniqueadvanced

Performs SVD on a context/next-token co-occurrence matrix implicit in next-token-prediction training, then reads off semantic structure from the sign pattern (orthant) of each embedding's coordinates in the singular-vector basis, showing increasingly fine-grained real semantic categories emerge as more singular components are included.

Used in (1 observation)

structure: Linear Subspace · models: GPT-2 Small · paper: Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations