A bespoke small GPT-2-style transformer (2-layer, 1-head), trained from scratch on in-context-learning tasks parameterized by a latent numerical/continuous concept, has add-k task vectors on a 1D linear manifold (PC1 > 99.9% of variance, offsets ordered, interpolation steers k); its Circular-Trajectory task vectors lie on a smooth low-dimensional manifold that the paper does not call linear

- Hong, Vasudeva, Sharan, Rashtchian, Raghavan & Panigrahy study whether transformers represent latent structure as part of in-context learning, and for tasks parameterized by a latent numerical concept, train a small (2-layer, 1-head, embedding dimension 128) GPT-2-style transformer from scratch. [hong-etal-2025-latent-concept-disentanglement] - Task vectors (layer-2 attention embeddings averaged over 200 sequences per task) of the add-k task lie on a 1D linear manifold (first principal component explains more than 99.9% of the variance, offsets ordered along the line), and interpolating along this line approximately interpolates the latent offset k; for the Circular-Trajectory task (latent radius) the task vectors lie on a smooth low-dimensional manifold in the first two principal components (93.7-97.1% variance across the three training-task counts, radii ordered), and steering with interpolated task vectors moves the output radius toward the target. The circular-trajectory geometry is described by the paper as a smooth manifold, not as a linear subspace. [hong-etal-2025-latent-concept-disentanglement]