methods / Theoretical / Analytical / Next-token-prediction SVD orthant decomposition
Next-token-prediction SVD orthant decomposition
Performs SVD on a context/next-token co-occurrence matrix implicit in next-token-prediction training, then reads off semantic structure from the sign pattern (orthant) of each embedding's coordinates in the singular-vector basis, showing increasingly fine-grained real semantic categories emerge as more singular components are included.