BOWS Toy Transformer
Research (Prieto, Stevinson, Barsbey, Birdal & Mediano 2026; custom 2-layer causal transformer, d_model=768, 8 heads, GELU FFN width 4*d_model, trained from scratch on WikiText-103 with a multi-hot 'seen-so-far' token-recovery objective)
Structures found in this family (2)
By model (1)
BOWS Toy Transformer (1 block, 8 heads, d_model=768)
Papers
From Data Statistics to Feature Geometry: How Correlations Shape Superposition (2026)