Causal attention without positions creates adjacency decay

measured in 1 paper

- In causal-attention transformers with no positional encodings, the self-cosine-similarity matrix shows a distance-decaying adjacency pattern: nearby positions are more similar than distant ones. [zuo-etal-2025-position-information-emerges-causal-transformers-similarity-nearby-embeddings] - An adjacency-probability score jumps from ~0.48-0.58 at the raw token-embedding layer to 0.80-0.99 after the first causal-attention layer, and appears even in randomly initialized models. [zuo-etal-2025-position-information-emerges-causal-transformers-similarity-nearby-embeddings] - The pattern is absent under non-causal attention (~0.5) and follows from causal prefix-averaging plus the near-orthogonality of high-dimensional embeddings. [zuo-etal-2025-position-information-emerges-causal-transformers-similarity-nearby-embeddings] - Trained NanoGPT models (6/12/24 layers, hidden 192/384/768) on four synthetic position-sensitive tasks; no pretrained LLMs. [zuo-etal-2025-position-information-emerges-causal-transformers-similarity-nearby-embeddings]

Structure

Anisotropy

Context

self-cosine-similarity matrix and the "adjacency pattern", adjacency probability score (row-wise rank-monotonicity statistic), pattern present in both randomly initialized and trained networks after one causal-attention layer, robust across layer count, hidden dimension, and typical initialization schemes, cosine-similarity-to-last-token outperforms embedding norm/variance as a position probe feature

Confirmed in models

NanoGPT Causal-NoPE, 6-layer (10.6M params) · NanoGPT Causal-Attention, No Positional Encoding (Causal-NoPE)

Method

Geometric analysis

Papers

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings — Zuo, Chunsheng, Guerzhoy, Pavel, Guerzhoy, Michael2025 · arXiv:2501.00073