A 1-layer transformer solves analogies via emergent vector arithmetic

measured in 1 paper

Minegishi et al. (ICML 2026 Spotlight) train a 1-layer, 1-head causal transformer (d=128) from scratch on a synthetic entity-relation analogy task [minegishi-etal-2026-emergent-analogical-reasoning] Analogical reasoning emerges once entity embeddings across categories become geometrically aligned, quantified by a measured decrease in Dirichlet energy during training [minegishi-etal-2026-emergent-analogical-reasoning] A Parallelism cosine-similarity metric between (e_t - e_s) and a learned functor direction f rises, indicating the model solves analogies via vector arithmetic e_t ~ e_s + f [minegishi-etal-2026-emergent-analogical-reasoning] The same Dirichlet-energy decrease appears layer-by-layer in a pretrained Llama via in-context learning, though the exact checkpoint is unconfirmed [minegishi-etal-2026-emergent-analogical-reasoning] The paper's nearest intervention (Appendix O) perturbs the data-level entity mapping rather than internal activations, so the claim is geometric, not causal [minegishi-etal-2026-emergent-analogical-reasoning]

Structure

Linear Direction

Context

vector-offset analogy solving in a trained transformer, Dirichlet energy as a geometric-alignment metric, Parallelism cosine-similarity metric, emergence during training, not assumed a priori

Confirmed in models

1-layer, 1-head causal Transformer (d_model=128, synthetic entity-relation analogy task) · Minegishi, Feng, Furuta et al. (2026) Analogical-Reasoning Toy Transformer

Method

Vector-offset analogy test, Geometric analysis

Papers

Emergent Analogical Reasoning in Transformers — Minegishi, Gouki, Feng, Jingyuan, Furuta, Hiroki, Kojima, Takeshi, Iwasawa, Yusuke, Matsuo, Yutaka2026 · arXiv:2602.01992