Mini-ICL toy transformers (custom, Yan, Yang & Zhong 2026)
University of Wisconsin-Madison / University of Chicago (custom-trained)
Structures found in this family (1)
By model (1)
Mini-ICL synthetic RoPE transformer (6L/2H/d128 for dice and Markov-chain tasks; 16L for linear regression) · small (6 or 16 layers, 2 heads, d=128), trained from scratch
Observations (1)
Papers
Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers (2026)