MATH · IN · MODELS

Vemula (2026) Custom Decoder-Only Transformers (FineWeb-Edu, from scratch)

Research (Vemula; custom-trained from scratch, arXiv:2606.20743)

Structures found in this family (1)

Dimensional collapse

By model (2)

Custom decoder-only transformer (160M, FineWeb-Edu, from scratch) · 160M
Dimensional collapse
Custom decoder-only transformer (290M, FineWeb-Edu, from scratch) · 290M
Dimensional collapse

Observations (1)

Massive activations re-emerge under a protected residual stream and resist a stronger sparsity penalty

Papers

Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test (2026)