Definition

Given four concept vectors $\mathbf{a},\mathbf{b},\mathbf{c},\mathbf{d} \in \mathbb{R}^d$ representing two instances of the same semantic relation (e.g. two country/capital pairs), a crystal face is the configuration in which $\mathbf{b}-\mathbf{a}$ and $\mathbf{d}-\mathbf{c}$ are proportional: $\mathbf{b}-\mathbf{a} = \lambda(\mathbf{d}-\mathbf{c})$ for some scalar $\lambda$. The special case $\lambda=1$ (the two edges exactly equal) is a genuine parallelogram: $\mathbf{b}-\mathbf{a}=\mathbf{d}-\mathbf{c}$, equivalently $\mathbf{a}+\mathbf{d}=\mathbf{b}+\mathbf{c}$, a single Linear Direction-like “function vector” transforming $\mathbf{a}\to\mathbf{b}$ and $\mathbf{c}\to\mathbf{d}$ identically.

Precision on the general case. $\lambda \neq 1$ gives a single shared direction but not a single shared displacement. It does not by itself yield a non-degenerate “trapezoid”: proportionality of two edges is compatible with self-intersection, coincident points, a fully collinear configuration, or (for $\lambda<0$) anti-parallel edges. Calling the $\lambda\neq1$ case a trapezoid requires the extra conditions of non-degeneracy and a fixed vertex order.

These four points are not a linear structure in $\mathbb{R}^d$. The crystal-face condition is an algebraic condition on a tuple $(\mathbf a,\mathbf b,\mathbf c,\mathbf d)\in(\mathbb{R}^d)^4$. The parallelogram set $\{(\mathbf a,\mathbf b,\mathbf c,\mathbf d): \mathbf b-\mathbf a=\mathbf d-\mathbf c\}$ is a genuine linear subspace of the configuration space $(\mathbb{R}^d)^4$ (kernel of $\mathbf a-\mathbf b-\mathbf c+\mathbf d$, dimension $3d$). The general proportional case with $\lambda$ free is instead the determinantal variety $\{\mathrm{rank}[\,\mathbf b-\mathbf a\ \ \mathbf d-\mathbf c\,]\le 1\}$ (all $2\times2$ minors vanish) — bilinear, generally not a linear subspace of $(\mathbb{R}^d)^4$.

A crystal is a larger cluster of many such faces (or higher-order tuples) sharing consistent edge directions, found by clustering pairwise difference vectors. (“Crystal” is used here as a coined term; a fully rigorous definition would fix its vertex/edge set, the admissible relations, the equivalence under translation/scale/rotation, and the criterion for merging quadruplets into one crystal.)

Intuition

The classical word2vec analogy $\vec{\text{king}}-\vec{\text{man}}+\vec{\text{woman}}\approx\vec{\text{queen}}$ (see Linear Representation Hypothesis‘s own parallelogram property) is the parallelogram special case. Requiring the exact same displacement for every instance of a relation is a strong demand — “Vienna is to Austria” and “Bern is to Switzerland” needn’t be separated by literally the same vector if, say, the two countries’ overall embeddings differ in unrelated ways (e.g. one name is longer). A trapezoid relaxes this to “the same direction, not necessarily the same magnitude” — still recovering one function vector per relation, but not demanding that unrelated per-instance differences vanish entirely.

Properties

• Strictly more general than a parallelogram, not a different claim. Every parallelogram is a trapezoid with $\lambda=1$; a trapezoid is the minimal relaxation that still yields exactly one shared function vector, rather than abandoning the linear-analogy structure altogether.

• Corrupted by distractor directions, not absent. Raw activation- or SAE-decoder-vector differences for a true semantic relation often fail to look like even a trapezoid, not because the relation isn’t linearly encoded, but because an unrelated, high-variance direction (e.g. token/word length) dominates the difference vector — projecting out such directions via a supervised signal-to-noise decomposition (e.g. Linear Discriminant Analysis (LDA), maximizing inter-cluster variance over intra-cluster variance) can turn a noisy quadruplet into a clean trapezoid.

• A crystal is a population of consistent faces, not one quadruplet in isolation. A single clean trapezoid could arise by chance from four points; a crystal’s evidential weight comes from many quadruplets across many relation instances sharing the same edge direction, found by clustering difference vectors (e.g. via k-means) rather than eyeballing one example.

• Function vectors are the crystal’s edges, not a separate object. The shared displacement/direction of a parallelogram or trapezoid face is exactly what the function-vector/task-vector literature extracts and uses for steering — crystal geometry and function-vector extraction describe the same underlying object from two different angles (a static geometric shape vs. an extracted, reusable intervention vector).

• One scale of a multi-scale structure, not the whole picture. Concept crystals describe the finest (“atomic”) scale of organization in a large feature point cloud; Feature Lobes (Spatial-Functional Modularity) describes coarser, functional-spatial clustering at an intermediate scale, and Intrinsic-dimension profile across depth-style depth-wise compression describes the largest (“galaxy”) scale — the three are complementary, not competing, descriptions of the same underlying representation space at different resolutions.

• Not limited to semantic analogies — the same shared-direction-per-relation shape recurs in syntax. Diego-Simón, D’Ascoli, Chemla, Lakretz & King (2024) find that dependency-parse edge vectors $s_{i,j}=h_i-h_j$ of the same Universal-Dependencies relation type (e.g. “nsubj,” “det”) are driven collinear by a contrastive probe, while different types are pushed orthogonal — exactly this page’s “crystal face” shape (a shared direction per relation), discovered via Polar probing (distance + angle) on real dependency-annotated sentences rather than via SAE dictionary clustering or word-analogy arithmetic. See the polar-syntax-relation-code Observation.

• The parallelogram special case has a first-principles derivation, not just a description. Korchinski, Karkada, Bahri & Wyart (2025) model words as vectors of independent binary semantic attributes, prove co-occurrence probability factorizes as a Kronecker product across attributes, and show the top eigenvectors of the (log) co-occurrence matrix are provably exactly the attribute vectors themselves ($\mathrm{rank}(\log M)\le d+1$) — from which $\mathbf{b}-\mathbf{a}=\mathbf{d}-\mathbf{c}$ (a parallelogram, $\lambda=1$) follows algebraically for any four words differing by the same attribute, rather than being an empirically-observed regularity in need of explanation. Validated against the measured Wikipedia PMI eigenvalue spectrum and shown analytically and empirically to survive deleting the analogy pairs from the training corpus (Co-occurrence statistics ablation). See parallelogram-analogy-structure-provably-emerges-from-kronecker-eigenvector-structure-of-independent-binary-attributes.

• Not limited to text — the same offset-vector structure recurs in acoustic word embeddings from speech models. Gauthier, Breiss, Leonard & Chang (2025) apply the classic Mikolov offset test (Vector-offset analogy test) directly to Wav2Vec 2.0 activations and a linear probe trained on them, finding regular English noun/verb inflections (“walk”→“walked”) recoverable via $\hat d=b-a+c$ at well above chance nearest-neighbor rank — but show the recovered direction tracks a broader distributional regularity across the lexicon rather than phonology or morphology specifically (it survives category/allomorph swaps but is fooled by same-surface-form “false friends”). See speech-ssm-representations-exhibit-global-linear-offset-geometry-linking-words-to-their-regular-inflected-forms.

• A second, mechanically independent first-principles derivation reaches the same parallelogram conclusion with no generative assumption about co-occurrence at all. Allen & Hospedales (2019) instead define “analogy” information-theoretically — a word $w^*$ “paraphrases” a word set $\mathcal{W}$ when the context distributions they induce have small KL divergence, and an analogy is the existence of one shared “transformation” (context-set addition/removal) realizing both $a\to a^*$ and $b\to b^*$ — and prove an exact identity decomposing any embedding-sum residual into named paraphrase-error and dependence-error terms, via Paraphrase-error decomposition. Unlike the Kronecker/spectral route above, no latent-attribute or low-rank generative structure for co-occurrence statistics is assumed; the identity holds for arbitrary conditional probability distributions, with the parallelogram’s approximation quality explained by how small a given analogy’s own error terms happen to be. See analogy-linearity-follows-exactly-from-a-paraphrase-based-definition-of-word-transformation.

• The two-edge crystal-face condition generalizes further to a many-family, regressed affine-combination law outside language, in a molecular foundation model. Soares, Shirasuna, Vital Brazil, Cerqueira, Zubarev & Schmidt (2024) construct six carbon-chain SMILES families and fit $\alpha\bar C_n^{CC}+\beta\bar C_k^i+B_0=\bar C_{n+k}^i$ (a longer-chain embedding as a learned affine combination of a base-chain embedding and a family-offset embedding) via linear regression on one held-out triple per family, then validate on the remaining 114 elements — reaching $R^2=0.99$ on SMI-TED289M embeddings versus only $R^2=0.55$ on a competing pretrained chemical-language model (MoLFormer) for the identical construction. Unlike the classical fixed $\lambda=1$ parallelogram or fixed-$\lambda$ trapezoid, here the composition coefficients ($\alpha,\beta,B_0$) are themselves fit rather than assumed, and the fitted held-out $R^2$ serves as a continuous, quantified measurement of how well the additive-composition shape holds — rather than a binary parallelogram/non-parallelogram judgment — providing a template for testing this family of shapes when the exact 1:1 edge-matching assumption is too strong for the domain. See soares-etal-2024-smi-ted-embeddings-satisfy-a-regressed-linear-additive-composition-law-across-molecular-carbon-chain-families.

• A closed-form condition for exactly when analogy parallelograms hold, and a confirmed distance-PMI linearity, both validated on real trained SGNS/GloVe vectors. Ethayarajh, Duvenaud & Hirst (2019) prove linear analogies hold exactly when a co-occurrence-shifted PMI (csPMI) value is constant across word pairs and the relevant vectors are coplanar, then confirm on real trained vectors that squared Euclidean distance is a linear function of -csPMI (Pearson r up to 0.514) and that per-category csPMI variance correlates with measured analogy accuracy (r=-0.70) — a formal condition with direct empirical confirmation, not just a description of when the shape “looks right.” See ethayarajh-etal-2019-linear-word-analogies-hold-exactly-when-a-shifted-pmi-value-is-constant-across-word-pairs-and-squared-euclidean-distance-is-a-linear-function-of-negative-shifted-pmi-confirmed-on-real-sgns-glove-vectors.

• A generative latent-discourse-random-walk model derives the parallelogram shape from first principles and confirms the prediction on real trained vectors. Arora, Li, Liang, Ma & Risteski (2016, RAND-WALK) model word co-occurrence as driven by a slowly-moving discourse vector under a log-linear generative law, derive analytically that this predicts analogy parallelograms and specific PMI-based similarity laws, and confirm both the predicted parallelogram structure and predicted word-frequency effects against real trained SGNS and GloVe vectors. See arora-etal-2016-rand-walk-model-derives-and-empirically-confirms-that-word-analogy-parallelograms-arise-from-a-latent-discourse-random-walk-generative-process-in-real-word2vec-glove-vectors.

Exercises

Base

1. If $\mathbf{b}-\mathbf{a} = \mathbf{d}-\mathbf{c}$ exactly, is the quadrilateral $\mathbf{a},\mathbf{b},\mathbf{d},\mathbf{c}$ (in that cyclic order) necessarily a parallelogram in the usual planar-geometry sense (opposite sides equal and parallel)? Justify briefly.

2. Why does $\lambda\neq1$ (a genuine trapezoid, not a parallelogram) still recover one function vector, rather than requiring two?

Middle

3. A raw quadruplet’s difference vectors are $\mathbf{b}-\mathbf{a}=(1,0,5)$ and $\mathbf{d}-\mathbf{c}=(1.02,-0.01,-5)$ in some 3D coordinate system where the third coordinate happens to encode an unrelated distractor (e.g. word length) with much larger scale than the first two. Explain why this quadruplet looks like neither a clean parallelogram nor a clean trapezoid before removing the third coordinate, and show it becomes (to good approximation) a parallelogram after projecting it out.

4. Explain, in terms of inter-cluster vs. intra-cluster variance, why a supervised projection (fit using knowledge of which quadruplets belong to which semantic relation) is necessary here, rather than plain unsupervised PCA on the difference vectors.

Pro

5. Suppose a dataset contains quadruplets from $R$ genuinely different semantic relations, each relation’s instances forming a near-perfect trapezoid only after distractor removal, but different relations’ function-vector directions are themselves highly correlated with each other (not orthogonal). What does this imply for using k-means to cluster all difference vectors together to discover crystals, versus clustering within each relation label separately, and why?