Synthetic GPT-2-style ICL transformers (custom, Yang, Lin, Lee, Papailiopoulos & Nowak 2025)

University of Wisconsin-Madison (custom-trained)

Structures found in this family (1)

By model (1)

Synthetic GPT-2-style ICL transformer (3-8 layers, trained from scratch on regression/token-offset/GINC/RegBench tasks) · small (3-8 layers, 2-8 heads, embedding dim 64-256), trained from scratch

Linear Direction

Observations (1)

Natural task vectors emerge weakly in deep models; a TVP loss forces a layer to carry the task

Papers

Task Vectors in In-Context Learning: Emergence, Formation, and Benefit (2025)