Translation Symmetry Hypothesis

structures / Hypotheses / Translation Symmetry Hypothesis

CLAIMhypothesisadvancedhow it's classified →

Representational manifolds (circles, open 1D continua, linearly decodable coordinates) share a common origin: translation symmetry in co-occurrence statistics. If co-occurrence depends only on distance along a semantic continuum, the co-occurrence matrix is circulant/Toeplitz and its eigenvectors are automatically Fourier modes.

Replicationcomputed from the corpus — never hand-assigned

1 paper2 architecture classes · one study1 domain2 model families · one study

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Statement

Let $M^\star_{ij}$ be a normalized co-occurrence statistic between items $i,j$ (e.g. a PMI-like quantity). Translation-symmetry assumption: for items belonging to a semantic continuum with coordinates $x_i,x_j \in \mathbb{R}^D$ ,

$M^\star_{ij} = C(\mathrm{dist}(x_i,x_j))$

i.e. the statistic depends only on the distance between items along the continuum, not on their absolute positions. A matrix with this property, restricted to a lattice, is circulant (if the underlying coordinate is periodic — a closed loop) or Toeplitz (if it is open) — and such matrices have eigenvectors that are exactly discrete Fourier / sinusoidal modes, regardless of the specific shape of the kernel $C$ .

Intuition

If how related two items feel depends only on how far apart they are along some shared axis — not on which specific pair you picked — then the algebra of “distance-only” matrices takes over and forces the eigenvectors into sinusoids, the same way a physical system with translation-invariant interactions (e.g. a uniform vibrating string or a periodic crystal lattice) is diagonalized by Fourier modes regardless of the details of the interaction strength function.

Properties

Circulant case (periodic coordinate) $\Rightarrow$ degenerate sin/cos eigenmode pairs. For a periodic lattice of size $n$ , a circulant matrix’s eigenvectors are $v_k[j] = e^{2\pi ijk/n}$ for $k=0,\ldots,n-1$ ; real eigenvectors pair up as $(\sin,\cos)$ combinations at each frequency, generically appearing with equal eigenvalues (degenerate pairs) — this is the mechanism proposed to underlie Circle (fundamental frequency) and would, if two independent periodic coordinates combine, underlie Torus.
Toeplitz case (open coordinate) $\Rightarrow$ quantized, non-degenerate sinusoidal modes. Without periodicity, boundary effects perturb the eigenmodes away from a simple $k_n=n\pi/L$ lattice into a self-consistent quantization condition, and the resulting eigenvalues are generically non-degenerate (no forced pairing) — the proposed mechanism behind 1D continuum manifold.
Slowest modes are linearly decodable from few components. Since the top eigenmodes (largest eigenvalues) are the lowest-frequency Fourier modes in either case, a handful of leading principal components already captures most of the coordinate’s variation — connecting this hypothesis to Linear Direction-style decodability of continuous coordinates from a low-rank probe.
Robust to targeted removal of specific entries. Because the predicted eigenmodes are a collective effect of many entries sharing the same underlying distance-dependence, removing a bounded number of individual entries of $M^\star$ (as opposed to changing the global kernel $C$ itself) generically leaves the dominant eigenmodes’ structure intact — a robustness property distinguishing “many items collectively carrying a shared latent variable” from “a few specific co-occurrences doing all the work.”
A generative mechanism, not a claimed shape itself. The hypothesis explains why circles and rippled 1D continua would arise (a mechanism, tied to corpus-statistics structure) — it is not itself a claimed geometric shape, which is why it lives among the hypothesis nodes rather than the structure nodes (Circle, 1D continuum manifold).

Relative to constructive-interference

This hypothesis explains why a co-occurrence/PMI matrix built from corpus statistics might already be circulant or Toeplitz (translation-invariant relatedness $\Rightarrow$ Fourier-mode eigenvectors) — a claim about the input statistics. Constructive Interference Hypothesis answers a different question: given such structure exists (in a feature covariance $\Sigma$ , not necessarily the same object as this hypothesis’s $M^\star$ ), when does a trained, bottlenecked, weight-decayed network’s own learned weights actually inherit those top eigenmodes, rather than default to a near-orthogonal packing that ignores the correlation entirely? The two mechanisms are complementary rather than competing: one supplies a reason the relevant matrix would have circular/Fourier structure, the other supplies the training-dynamics reason a compressed representation would surface that structure instead of overriding it.

Exercises

Base

Write down the $4\times4$ circulant matrix with first row $(2,1,0,1)$ (i.e. each subsequent row is a cyclic shift of the previous one).

Solution

$\begin{pmatrix}2&1&0&1\\1&2&1&0\\0&1&2&1\\1&0&1&2\end{pmatrix}$

Each row is the previous row shifted one position to the right, with wraparound — the defining property of a circulant matrix.

Is the matrix from Exercise 1 also Toeplitz (constant along diagonals)? Is every circulant matrix Toeplitz?

Solution

Yes, it is Toeplitz (check: every diagonal, going down-right, has constant entries — e.g. the main diagonal is all $2$ ‘s, the first superdiagonal is all $1$ ‘s where present). In general, every circulant matrix is a special case of a Toeplitz matrix (one where the diagonals additionally “wrap around” consistently with the rest of the pattern) — circulant $\Rightarrow$ Toeplitz, but not conversely: a generic Toeplitz matrix (e.g. from an open, non-periodic kernel) need not have the wraparound consistency that makes it circulant.

Middle

Verify that $v_k[j] = e^{2\pi ijk/n}$ is an eigenvector of the circulant matrix from Exercise 1 (with $n=4$ ) for $k=0$ , and find its eigenvalue.

Solution

For $k=0$ : $v_0 = (1,1,1,1)$ (constant vector). Multiplying by the matrix: row 1 gives $2+1+0+1=4$ ; row 2 gives $1+2+1+0=4$ ; similarly rows 3,4 give $4$ . So $Mv_0 = 4v_0 = (4,4,4,4)$ , confirming $v_0$ is an eigenvector with eigenvalue $4$ — which equals the row sum, as expected for the constant eigenvector of any circulant (or more generally any matrix with constant row sums).

For a general circulant matrix with first row $(c_0,c_1,\ldots,c_{n-1})$ , derive the eigenvalue corresponding to eigenvector $v_k[j]=e^{2\pi ijk/n}$ , in terms of the $c_m$ ‘s and $k$ .

Solution

The circulant matrix has entries $M_{jl} = c_{(l-j) \bmod n}$ . Then $(Mv_k)_j = \sum_l M_{jl}v_k[l] = \sum_l c_{(l-j)\bmod n}\, e^{2\pi ilk/n}$ . Substituting $m=(l-j)\bmod n$ , i.e. $l = j+m \pmod n$ : $=\sum_m c_m\, e^{2\pi i(j+m)k/n} = e^{2\pi ijk/n}\sum_m c_m e^{2\pi imk/n} = v_k[j] \cdot \Big(\sum_{m=0}^{n-1} c_m e^{2\pi imk/n}\Big)$ . So $v_k$ is indeed an eigenvector, with eigenvalue $\lambda_k = \sum_{m=0}^{n-1}c_m e^{2\pi imk/n}$ — the discrete Fourier transform of the first row, evaluated at frequency $k$ . This holds for any choice of $c_0,\ldots,c_{n-1}$ (any kernel $C$ ), which is exactly the sense in which “the eigenvectors are automatically Fourier modes regardless of the specific shape of the kernel.”

Pro

Show that for a real, symmetric circulant matrix ( $c_m = c_{n-m}$ , i.e. the kernel depends only on the cyclic distance $\min(m,n-m)$ ), the eigenvalues $\lambda_k$ and $\lambda_{n-k}$ coincide, producing the degenerate pairing referenced in the Properties section.

Solution

From Exercise 4, $\lambda_{n-k} = \sum_m c_m e^{2\pi im(n-k)/n} = \sum_m c_m e^{2\pi im}e^{-2\pi imk/n} = \sum_m c_m e^{-2\pi imk/n}$ (using $e^{2\pi im}=1$ for integer $m$ ) $= \overline{\lambda_k}$ (complex conjugate of $\lambda_k$ , since conjugating $e^{2\pi imk/n}$ gives $e^{-2\pi imk/n}$ and the $c_m$ are real). But $\lambda_k$ is already real whenever the $c_m$ satisfy the symmetry $c_m=c_{n-m}$ (a direct computation, or the general fact that a real symmetric circulant has a real spectrum) — so $\lambda_{n-k}=\overline{\lambda_k}=\lambda_k$ . Since $v_k$ and $v_{n-k}$ are complex conjugate eigenvectors with the same real eigenvalue, their real and imaginary parts (proportional to $\cos$ and $\sin$ of the same frequency) span a 2-dimensional eigenspace at that shared eigenvalue — the degenerate sin/cos pair.

A Toeplitz (non-circulant) matrix has entries $M_{jl} = C(|j-l|)$ for $j,l \in \{0,\ldots,n-1\}$ , with no wraparound. Explain concretely, via a small $3\times3$ example, why $v_k[j]=e^{2\pi ijk/n}$ generally fails to be an exact eigenvector in this case, in terms of the “missing” wraparound terms that the circulant computation in Exercise 4 relied on.

Solution

Take $n=3$ , kernel $C(0)=2,C(1)=1,C(2)=0$ (say the kernel vanishes beyond distance 1), giving Toeplitz matrix $M = \begin{pmatrix}2&1&0\\1&2&1\\0&1&2\end{pmatrix}$ (note $M_{02}=C(2)=0$ , whereas a circulant version would instead wrap around and use $C(1)=1$ there, i.e. treat index-distance $2$ on a length-3 cycle as equivalent to distance $1$ the other way). Testing $v_1[j]=e^{2\pi ij/3}$ : the circulant eigenvector derivation in Exercise 4 relied on every term $c_{(l-j)\bmod n}$ appearing with the cyclic distance, but the actual Toeplitz matrix uses $C(|j-l|)$ (non-cyclic distance) — for the pair $(j,l)=(0,2)$ , the Toeplitz matrix has $C(2)=0$ while the circulant computation implicitly would have used $C(1)=1$ (since $2 \equiv -1 \pmod 3$ ). This mismatch at the boundary (row 0’s interaction with the far end, and symmetrically row $n-1$ ‘s interaction with the near end) is exactly the “missing wraparound terms” — direct multiplication $Mv_1$ will not come out proportional to $v_1$ in general because these edge entries break the clean cyclic-shift structure the circulant proof depended on. This is the algebraic origin of the “boundary effects perturb the eigenmodes away from a simple lattice” property claimed for the open (Toeplitz) case.

Found in (0 observations · 0 families)

No observations confirm this structure yet.