Correlated-feature packing under a tight bottleneck reproduces circular geometry

Prieto et al. introduce Bag-of-Words Superposition (BOWS): tied-weight linear/ReLU autoencoders (W in R^{m x d}) reconstructing binary bag-of-words vectors of WikiText-103/OpenWebText, vocabulary d=10,000, latent widths m 200-1000 [prieto-etal-2026] They derive that a linear autoencoder's loss-optimal W^T W is the orthogonal projector onto the feature covariance's top-m principal components, making interference proportional to a feature's own signal rather than adversarial noise when Sigma is near-low-rank [prieto-etal-2026] A weight-norm identity (||W||_F^2 = m in the constructive regime vs ~d in the classical near-orthogonal ReLU-filtered regime) shows weight decay favors constructive packing whenever m << d [prieto-etal-2026] Under a tight bottleneck (m<6), autoencoder weight columns recover a clean circular arrangement of month-words tracking Sigma's top-2 eigenvectors (linear-decoder R^2=0.98+/-0.00015) [prieto-etal-2026] Under strong compression (m=200) or explicit weight decay, weight columns form semantically coherent clusters (verbs, proper names, sports); this clustering weakens as m grows toward 800 and reappears when weight decay is added [prieto-etal-2026] A custom single-block causal toy transformer (one encoder block, 8 heads, d_model=768, vocabulary 16,000) trained on a seen-so-far multi-hot token-recovery objective reproduces the same clusters and circular structure in its residual stream [prieto-etal-2026] The paper stresses this constructive-packing mechanism is distinct from value-coding (a single computed quantity linearly encoded), since both can yield circular geometry but only the former evidences constructive interference [prieto-etal-2026]