Positional encodings anchor index-based spatial structure in ViTs

- In a ViT-S/16 trained from scratch on ImageNet-100, no-PE models build content-driven spatial structure, whereas every positional-encoding scheme shifts representations to an index-anchored organization stable under token permutation (SSDC ~0 for no-PE vs 0.3-0.47 for PE). [mannes-2026-positional-encodings-anchor-spatial-structure-vits] - Robustness tracks the integrity of this positional reference frame, not the encoding type: JPEG-q5 fragility is ~0.30 (RoPE), ~0.43 (absolute/sinusoidal) and ~0.63-0.66 (no-PE or permutation-trained). [mannes-2026-positional-encodings-anchor-spatial-structure-vits] - Scaling positional-embedding magnitude down removes the index anchoring and raises fragility in lockstep; the authors frame this as intervention-based evidence, not a fully isolated causal mechanism. [mannes-2026-positional-encodings-anchor-spatial-structure-vits] - Single model: ViT-S/16 (~22M params) with absolute, sinusoidal, rotary and no-PE variants. [mannes-2026-positional-encodings-anchor-spatial-structure-vits]