Structures — Math in LLMs Research Map

MATH · IN · MODELS

Structures

The map's objects are classified by their mathematical role (the type axis), not by a single parent category. Orthogonal properties (curvature, topology, linearity, specification) are facets shown on each node's page. See the classification tree for the full road from an object to its leaf.

Hypotheses are claims about objects rather than objects, so they live on their own page —all 14 of them.

Each card carries the same computed replication read-out as the structure's own page:filled — two or more papers with no shared author reported it (replication);outlined — two or more values, but all from one study (breadth);grey — a single value. Derived from the corpus, never hand-assigned.

geometric-object

A set the activations lie on (carrier / subset / embedded set / direction).

1D continuum manifold

Open (non-periodic) one-dimensional manifold with extrinsic 'ripples' encoding an ordered continuous coordinate. Projected onto any two principal components it traces a Lissajous curve; higher-frequency harmonics are the ripples.

14parch 4dom 4fam 1410 obs.

2D Grid (Square Lattice)

A finite 2-dimensional square-lattice arrangement of points — items placed at integer grid coordinates (i,j) with horizontal/vertical nearest-neighbour adjacency. Distinguished from a torus by being bounded and non-periodic (contractible, open boundary; no global wrap-around cycle vs. H1=Z^2), and from a 1D continuum by having two independent lattice coordinates rather than one.

1parch 1dom 1fam 21 obs.

Affine Subspace

Linear subspace with an offset: A = x₀ + V. Flat, open manifold of zero curvature. Unlike a linear subspace, need not pass through the origin.

6parch 5dom 2fam 106 obs.

Closed 1-manifold S¹ encoding a single cyclic variable. Topologically distinct from the torus T² = S¹ × S¹: one independent cycle (H₁ = ℤ), not two (H₁ = ℤ²).

33parch 9dom 5fam 3327 obs.

The topological cone construction over a base space B — rays from an apex through every point of B, scaled by a shared non-negative 'magnitude' coordinate. Two instances: a polyhedral cone over a finite set of directions, and a circular cone over S¹ (angle × magnitude).

10parch 3dom 2fam 107 obs.

Decision boundary (as a codimension-1 hypersurface)

The codimension-1 level set $\{x : f(x) = 0\}$ separating two predicted classes in a classifier's input (or feature) space — a distinct geometric object at the function level, as opposed to the activation-cloud/representation-level manifolds this map otherwise catalogs.

7parch 3dom 3fam 137 obs.

Generalized Helix

A scalar quantity a is embedded as helix(a) = C·B(a), where B(a) stacks a linear term with cos/sin pairs at several periods T_1,...,T_k — a single number simultaneously encoded as 'how far along an unbounded axis' and 'where on each of several superposed circles,' generalizing a plain circle (one period) to several periods with generally incommensurate ratios, embedded into a linear subspace by a single learned matrix.

2parch 1dom 1fam 32 obs.

Hyperbolic Manifold

Manifold of constant negative curvature. Ball volume grows exponentially with radius — a natural geometry for encoding hierarchies with exponential branching.

2parch 3dom 2fam 62 obs.

Laguerre-Voronoi partition (weighted power diagram of a linear readout layer)

The weighted Voronoi (Laguerre / power-diagram) partition of representation space induced by a linear readout layer's own weights and biases: each output unit $j$ becomes a center $c_j$ with a weight $\nu_j$ derived from its bias, and the cell $\mathcal{C}_j=\{z: \|z-c_j\|^2-\nu_j \le \|z-c_i\|^2-\nu_i\ \forall i\}$ is a convex polyhedron algebraically identical to the layer's own argmax decision region. A 'concept' is redefined as an entire cell (a region), not a point or a direction, and a 'category' as a union of cells.

1parch 1dom 1fam 41 obs.

Linear Direction

Unit vector rᶠ ∈ ℝᵈ encoding feature f. The projection rᶠ · x monotonically reflects the feature's value in the context defining activation x.

288parch 14dom 15fam 159283 obs.

Linear region arrangement (polyhedral tessellation of input space)

The partition of a piecewise-linear (e.g. ReLU) network's input space into finitely many convex polyhedral cells — one per activation pattern — on which the network computes a single affine map. A distinct object from [[decision-boundary]]: the boundary is one codimension-1 level set of the *output*, while the region arrangement is the *entire* combinatorial/geometric tessellation induced by every breakpoint of every unit, at every layer.

3parch 2dom 2fam 23 obs.

Linear Subspace

k-dimensional linear subspace V ⊂ ℝᵈ corresponding to a group of related features. Generalises the single feature direction to the multidimensional case.

139parch 14dom 11fam 106140 obs.

Lissajous Curves

A parametric curve traced by superposing sinusoids of different frequencies along different axes: (x(t), y(t)) = (A sin(at+δ₁), B sin(bt+δ₂)). The 2D shadow of any multi-frequency 1D manifold — the shared geometric signature behind manifolds-circle/torus and manifolds-1d-continuum.

6parch 3dom 2fam 84 obs.

Local hyperboloid-type quadric patch

A local patch of embedding space, spanned by a tightly-controlled family of near-paraphrase sentences, that is better fit by a curved implicit quadric surface (predominantly one-sheeted-hyperboloid type) than by an affine subspace or ellipsoid — an open, saddle-like local carrier rather than a flat cluster or a closed convex blob.

1parch 1dom 1fam 21 obs.

Minkowski Sum of Tile Polytopes (Minkowski Representation Hypothesis)

The whole activation space of a layer, not just one categorical concept, decomposes as the Minkowski (vector) sum of many tile polytopes — disjoint convex hulls over partitioned archetype dictionaries — with any activation expressed as a block-sparse convex combination using only a handful of active tiles at once. Not a relativistic/indefinite-metric structure despite the name: 'Minkowski' here means the classical convex-geometry Minkowski sum A ⊕ B = {a+b : a∈A, b∈B}.

1parch 1dom 1fam 11 obs.

Paraboloid (circular × pinched continuum)

A surface of revolution combining a circular coordinate with a continuous 'axial' coordinate, where the circle's radius (spread) varies with — and can shrink toward zero at — position along the axis, unlike a flat cylinder (S¹ × ℝ) where the circle's radius stays constant.

1parch 1dom 1fam 12 obs.

Pinched Manifold (Singular Quotient)

A word/embedding space is not a smooth manifold but a pinched one: a manifold of 'meanings' with some of its points identified (glued together) to produce singular points, each singularity corresponding to a polysemous word with multiple senses folded into one vector.

1parch 1dom 1fam 11 obs.

Polytope (Simplex)

The convex hull of k vector representations, one per value of a mutually-exclusive (categorical) concept — e.g. {mammal, bird, reptile, fish}. Generically a (k-1)-simplex, not just k independent directions: unlike a binary feature (one direction), a categorical feature needs a magnitude-bearing vector per value so that differences between values remain meaningful.

21parch 8dom 4fam 2119 obs.

Closed homogeneous manifold of constant positive curvature. Normalisation onto a fixed norm places activations on the unit hypersphere Sⁿ⁻¹, making cosine similarity the natural distance.

4parch 2dom 2fam 64 obs.

Stratified Manifold with Continuous Fibers

A representation space partitioned into a discrete set of macroscopic basins (indexed by a discrete output/class variable) with each basin's interior further organized by continuous 1D fibers that extend across basin boundaries, tracking a separate continuous quantity — a discrete clustering structure and a continuous stratification structure coexisting at different scales of the same space, rather than either alone.

1parch 2dom 2fam 21 obs.

Closed manifold T² = S¹ × S¹ — the product of two independent circles. Distinguished from a bare circle by first homology: H₁(T²) = ℤ² (two independent cycles) versus H₁(S¹) = ℤ (one).

9parch 2dom 3fam 87 obs.

configuration

An ordered tuple of several points — a relational pattern in configuration space.

Attention reference frame (sink-token anchor configuration)

A small, distinguished set of token positions R = {p_1,...,p_k} whose key vectors serve as fixed geometric anchors that every other token's query orients against, formalized as R = (M, P, phi): a coordinate chart on the representation manifold M. The attention-sink phenomenon (disproportionate attention to specific tokens) is the attention-weight signature of this configuration, not an architectural artifact.

1parch 2dom 1fam 81 obs.

Concept Crystals (Parallelogram/Trapezoid Structure)

A quadruplet of concept vectors (a, b, c, d) whose difference vectors are proportional, b-a ∝ d-c — a parallelogram when the two edges are exactly equal (one shared 'function vector'), a trapezoid when only proportional — generalizing the classical man:woman::king:queen analogy into a broader family of shapes built from repeated linear-direction edges.

10parch 4dom 3fam 1110 obs.

Relation frame (ordered multi-token tuple geometry)

A finite, ordered configuration of several token hidden states jointly bound by one relation — described not by a single direction or one fixed subspace, but by the tuple's centroid, centered shape, spanned subspace, and orientation (sign pattern of its Plücker/determinant minors).

1parch 1dom 1fam 11 obs.

operator

A map or matrix acting on the cloud (not a shape of the cloud).

Attribute-Induced Embedding Folding

A continuous, attribute-indexed vector field on an embedding manifold — the pushforward through the embedding map of an attribute's local one-parameter variation. A model achieves invariance to an attribute not by ignoring it but by nonlinearly folding/contracting the space along this field; the amount of folding (an 'invariance energy') is a per-attribute, per-architecture quantity, not a fixed geometric constant.

1parch 1dom 1fam 11 obs.

Conceptor (soft ellipsoidal region)

A soft generalization of a linear subspace: a positive semi-definite matrix C, fit to the covariance of a set of activation vectors, whose eigenvalues lie continuously in [0,1] rather than being restricted to exactly 0 or 1 — so 'membership' in the region is graded rather than binary.

3parch 2dom 2fam 53 obs.

Continuous causal attention operator (CCT)

A generalization of discrete causal self-attention that replaces the sum over token positions with an integral over a continuous time variable, applied to a continuous-time embedding function x(t) rather than a discrete token sequence — using the same pretrained weight matrices unchanged, and proven to reduce exactly to standard discrete attention when the continuous function is stepwise-constant with unit-duration steps.

1parch 1dom 1fam 51 obs.

Finite-Lag Transport Tensor

An empirically-estimated tensor G_delta summarizing how a recurrent network's own hidden-state trajectory transports over a fixed lag delta -- decomposing exactly into how much successor states disperse around their conditional mean (conditional-spread) and how far that mean itself moves (coherent-displacement), plus a rotational/circulation term -- distinct from a Jacobian-based linearization since it is a statistic of realized (source, successor) state pairs, not a local derivative.

1parch 1dom 1fam 11 obs.

Jacobian non-normality (Schur decomposition of per-layer operators)

The Jacobian J_l of a transformer block's output with respect to its input (in the residual stream) is, at every layer of every trained model tested, a non-normal operator: its complex Schur decomposition J = Q(Lambda+N)Q* has a nonzero off-diagonal part N distinct from its eigenvalues, and training reshapes how non-normal each layer's Jacobian is (a depth-wise gradient from rotation-dominated to near-symmetric), with N -- not the eigenvalue spectrum Lambda -- responsible for a cumulative, depth-compounding collapse in the effective rank of composed Jacobians.

1parch 1dom 1fam 31 obs.

Symmetric/skew decomposition of the attention query-key operator

The combined query-key bilinear form W_qk = W_Q W_K^T of a trained attention head decomposes uniquely into a symmetric part (undirected, content-based similarity) and a skew-symmetric part (directed, positional/causal asymmetry); real trained encoder-only Transformers are measurably symmetric-dominant and real trained decoder-only Transformers are measurably directionality-dominant, and imposing the symmetric structure at initialization causally speeds convergence.

1parch 1dom 1fam 11 obs.

metric-model

A metric, seminorm, or learned distance model.

Diffusion Spacetime Information Geometry

A diffusion model's own family of denoising distributions {p(x0|xt,t)}, indexed jointly by noisy state and time, forms an exponential-family statistical manifold carrying a Fisher-Rao metric; geodesics under this metric (computable in closed form from the trained denoiser, with no numerical simulation) define a Diffusion Edit Distance between clean data points, distinct in kind from any Riemannian metric obtained by pulling back an ambient/activation-space metric through the model's Jacobian.

1parch 1dom 2fam 21 obs.

Score-Jacobian pullback Riemannian metric

A genuine Riemannian metric tensor G = JᵀJ built directly from the Jacobian of a real diffusion model's own score function, with no linearity assumption and no ambient/latent metric pulled back through an assumed linear structure — used to compute length-minimizing geodesics between real data points for interpolation. Distinct from a Fisher-Rao information-geometric metric (diffusion-spacetime-information-geometry) despite both living on a diffusion model's own manifold.

1parch 1dom 1fam 21 obs.

Tree Metric Embedding

A learned inner product (equivalently, a linear transform B) under which squared distance between two representations approximates graph distance in a discrete tree, and squared norm approximates depth from the root — embedding an entire tree's edge structure into the metric of a linear subspace, not just a single decodable relation.

14parch 4dom 2fam 1512 obs.

distribution-property

An unsupervised property of the activation distribution.

A representation space is anisotropic if activation vectors are not directionally uniform — they occupy a narrow angular cone rather than pointing every which way, so that two randomly chosen vectors have unexpectedly high cosine similarity on average. Origin: Ethayarajh (2019), building on Mimno & Thompson (2017)'s term for static skip-gram embeddings.

33parch 11dom 3fam 4433 obs.

Dimensional collapse

A representation space exhibits dimensional collapse if the embedding covariance matrix's singular-value spectrum decays toward zero well before the ambient dimension, so the population of embedding vectors spans a genuine lower-dimensional linear subspace rather than the full available space — a rank-of-the-cloud property, distinct from anisotropy's directional-concentration property of the same cloud.

16parch 6dom 4fam 2416 obs.

Task-induced symmetry class of a learned representation

Which symmetry group (invariance class) a trained agent's representation becomes invariant to is not fixed by the environment alone — it is empirically discovered to depend on the training objective. A distribution-property claim about the representation's response to input transformations, distinct from any single measured metric (curvature, dimension, anisotropy) of the representation's static shape.

1parch 1dom 1fam 11 obs.

labeled-data-property

A property of labeled sets (separability, margin, decodability).

Linear Separability

Two activation classes are linearly separable if a hyperplane can separate them — the necessary condition for a clean linear feature direction to exist.

66parch 11dom 11fam 6565 obs.

Nonlinear World-Model Decodability

A discrete world-state (e.g. a board configuration) is reliably decodable from a network's internal activations only by a nonlinear probe (a small MLP), not a linear one — the representation exists and is causally load-bearing, but the labeled classes it induces are not linearly separable in the model's own basis.

2parch 2dom 2fam 32 obs.

measurement

A computed quantity or profile (scalar / vector / functional).

Attention-graph spectral profile (Fiedler value, HFER, smoothness, spectral entropy)

Treating a transformer's attention matrix at each layer as the weighted adjacency matrix of a dynamic graph over tokens, and reducing it to four scalar diagnostics from its graph Laplacian spectrum: the Fiedler value (algebraic connectivity), high-frequency energy ratio (HFER), graph-signal smoothness (Dirichlet energy of hidden states as a graph signal), and spectral entropy of the Laplacian eigenvalue distribution.

1parch 1dom 1fam 41 obs.

Attention-weight frequency-band specialization (DFT/wavelet decomposition of attention as a position-indexed signal)

Treating each attention head's weight pattern, at a fixed query position, as a 1-D signal indexed by relative key position, and decomposing it via DFT or wavelet transform into frequency bands -- revealing that individual heads specialize in distinct, consistent frequency bands (e.g. low-frequency/long-range vs. high-frequency/local) rather than an undifferentiated graph-connectivity pattern.

1parch 1dom 1fam 21 obs.

Curvature profile of the representation manifold

How much a network's representation manifold bends — measured, not assumed as a named shape. Two non-interchangeable notions: extrinsic principal curvature (MAPC, local-PCA fit, ambient bending) and intrinsic graph-Ricci curvature (Ollivier/Forman, optimal-transport/combinatorial). Reported as a profile across depth/training, orthogonal to intrinsic dimension.

18parch 10dom 6fam 1718 obs.

Intrinsic-dimension profile across depth

The dimensionality of the data manifold a network's own representations lie on is not fixed — it expands sharply in early layers, then contracts to a low-dimensional plateau or minimum, and the layer at that minimum tends to carry the most abstract, task-relevant semantic content.

38parch 11dom 8fam 4236 obs.

Persistent-homology / Betti profile across depth

Betti numbers / persistent topological features (independent loops, holes, connected components) of a real model's activation point-cloud, tracked as a profile across layers, training, or model comparison — a computed topological-data-analysis quantity, not a claim that the representation manifold is any specific named shape. Orthogonal to intrinsic dimension and to curvature: a topologically rich point cloud can be low- or high-dimensional, flat or curved.

3parch 3dom 3fam 43 obs.

Representational-similarity trajectory across depth

How similar a network's own layer-wise representations are to each other (via a similarity metric such as CCA, SVCCA, or CKA) traces a specific, measurable trajectory across depth — layers cluster into similarity-based stages, and the trajectory's shape shifts with training objective, architecture, or between independently trained networks.

8parch 7dom 3fam 168 obs.

Semantic information production rate along the diffusion trajectory

The Bayes-optimal conditional entropy of a semantic class label given a diffusion model's own noisy state, and its time-derivative (bits of class information produced per unit diffusion time), traces a measurable, non-uniform trajectory across the generative process -- a purely information-theoretic diagnostic of when a real trained diffusion model's generative dynamics commit to specific semantic content, distinct from any curvature or dimensionality measurement of the same trajectory.

1parch 1dom 1fam 11 obs.

Spatial-frequency (spectral) accessibility profile across depth

How linearly recoverable each spatial-frequency band of the original input image is from a vision model's own layer-wise representation, tracked as a profile across depth via ridge-regression probes against ground-truth Fourier-energy targets, relative to a dimension-matched random-projection baseline that isolates learned-transformation effects from pure dimensionality change.

1parch 2dom 2fam 21 obs.

dynamical-object

An object defined relative to a dynamical system (attractor, invariant set).

Ghost Point (saddle-node bifurcation remnant)

A transient dynamical bottleneck in a recurrent network's state space — not itself a fixed point, but a remnant of one that vanished via a saddle-node bifurcation earlier in training — identified via a near-zero eigenvalue of the network's speed-function Jacobian, and shown to slow learning by trapping the trajectory during training.

1parch 1dom 1fam 11 obs.

Limit cycle (stable periodic attractor)

A closed, contractive periodic orbit that a recurrent network's own dynamics settle into and recover to after perturbation — a property of the update rule itself (established via finite-time-Lyapunov contractivity and recovery-after-perturbation), not a static shape traced by activations across many different inputs. A dynamical sibling of the line attractor: same role (a property of the dynamics), different topology (closed loop of periodic states vs. an open line of marginally-stable fixed points).

1parch 2dom 1fam 11 obs.

A 1D manifold of marginally-stable fixed points of a recurrent network's own update rule, along which the network's state moves to integrate (accumulate) scalar evidence over time — distinct from a static 1D manifold traced by activations across many different inputs, since a line attractor is a property of the dynamics itself, established by finding fixed points and confirming they are attracting and marginally stable along one direction.

6parch 2dom 3fam 55 obs.

combinatorial-object

An order, lattice, graph, matroid, or complex.

Concept lattice (Formal Concept Analysis half-space model)

Each attribute is a linear direction defining an origin-passing half-space; a concept (a set of co-occurring attributes) is the polyhedral cone formed by intersecting those half-spaces. The set of all such concepts, ordered by inclusion, forms a complete lattice with well-defined meet (intersection) and join (least common generalization) operations — recovering Formal Concept Analysis directly from embedding geometry.

2parch 2dom 1fam 42 obs.

empirical-pattern

An observed organization of the data in a specific model.

Concept Cluster Heterogeneity

A nominal concept class's activations, discovered via unsupervised clustering (k-means/GMM) of real hidden states, form multiple distinguishable sub-clusters rather than a single unimodal blob around one class mean — each sub-cluster corresponding to a semantically coherent subgroup (e.g. distinct instruction types within a 'harmful' class).

3parch 3dom 2fam 63 obs.

Feature Lobes (Spatial-Functional Modularity)

A partition of a large feature point cloud into spatially-coherent regions ('lobes'), discovered purely from functional co-occurrence statistics with no positional information used, that nonetheless turn out to occupy distinguishable spatial regions — an intermediate-scale organizing principle between single-direction/subspace structure and the point cloud's global shape.

1parch 1dom 1fam 11 obs.