In-context exemplars reorganize concept geometry to a specified graph

measured in 1 paper

Park et al. (ICLR 2025) define an arbitrary graph (grid or ring) over semantically unrelated concept tokens in-context and feed random-walk traces over it as exemplars [park-etal-2025-icl-representations] As context length scales, windowed activations projected onto top principal components suddenly reproduce the specified graph's topology rather than the pretrained semantic clustering of those tokens [park-etal-2025-icl-representations] A Dirichlet-energy metric against the ground-truth graph decreases with context size, quantifying the sudden reorganization, with a spectral-embedding argument linking it to the top PCA components [park-etal-2025-icl-representations] When reference concepts already carry correlated pretrained semantics (e.g. weekdays), the in-context graph appears only in PC3-4 rather than overriding the pretrained ring in PC1-2 [park-etal-2025-icl-representations] Intervening on concept projections along the identified principal components changes next-node prediction accuracy, a causal check beyond the PCA visualization (exact effect sizes not captured here) [park-etal-2025-icl-representations] Tested mainly on Llama-3.1-8B and replicated on Llama-3.2-1B, Llama-3.1-8B-Instruct, Gemma-2-2B and Gemma-2-9B [park-etal-2025-icl-representations]

Structure

Circle, 2D Grid (Square Lattice)

Context

in-context learning, conceptual role semantics, graph tracing, Dirichlet energy, phase transition, competing pretrained vs. in-context structure, causal PCA intervention

Confirmed in models

Llama-3.1-8B · Llama, Llama-3.2-1B · Llama, Llama-3.1-8B-Instruct · Llama, Gemma-2-2B · Gemma, Gemma-2-9B · Gemma

Method

PCA, Causal interventions (steering)

Papers

In-Context Learning of Representations — Park, Core Francisco, Lee, Andrew, Lubana, Ekdeep Singh, Yang, Yongyi, Okawa, Maya, Nishi, Kento, Wattenberg, Martin, Tanaka, Hidenori2025 · arXiv:2501.00070