Definition

A recurrent network with update rule $\mathbf{h}_{t+1}=F(\mathbf{h}_t,\mathbf{x}_t)$ exhibits an (approximate) line attractor when: (1) its zero-input fixed points $\{\mathbf{h}_i^*\}$ (found via Fixed-point dynamics analysis) lie along an approximately 1-dimensional manifold in state space, parameterized by a coordinate $\theta$; (2) each fixed point is attracting — nearby states converge to it under zero input; and (3) each fixed point is marginally stable along the manifold’s own direction — the top eigenvalue of the local Jacobian $\mathbf{J}^{\mathrm{rec}}$ satisfies $\lambda_1\approx1$ (neither decaying nor diverging), with its right eigenvector $\mathbf{r}_1$ aligned to the manifold’s principal direction $\mathbf{m}$. An input that carries evidence for the integrated quantity has an instantaneous effect $\mathbf{J}^{\mathrm{inp}}\mathbf{x}$ that, projected onto the left eigenvector $\boldsymbol\ell_1$, pushes the state along $\mathbf{r}_1$ — moving it to a neighboring point further along the line — while uninformative input has near-zero projection and leaves the state essentially unmoved.

Intuition

A marble resting in a long, shallow, nearly-flat groove (rather than a single well): pushing it sideways, out of the groove, it rolls straight back (attracting); but pushing it along the groove, it stays wherever it’s pushed to rather than rolling back to a single point (marginal stability along that one direction) — exactly the mechanism needed to accumulate a running scalar tally (e.g. net positive-minus-negative sentiment evidence) without either forgetting it or blowing up.

Relative to a static 1D continuum manifold

1D continuum manifold describes a 1D curve traced out by activations across many different inputs/contexts at a single point in computation — a purely descriptive, static shape. A line attractor is instead a claim about the network’s own update dynamics: the manifold is made of fixed points of the recurrence itself, and the claim includes stability properties (attracting, marginally stable) that a static point-cloud shape has no analogue of. The two can coincide geometrically (both may look like a 1D curve in a PCA projection) while being evidentially distinct claims — one about where activations sit, the other about how the system’s own dynamics behave near where they sit.

Key evidence

Maheswaranathan, Williams, Golub, Ganguli & Sussillo (NeurIPS 2019, arXiv:1906.10720) find this structure across four RNN architectures (LSTM, GRU, UGRNN, vanilla RNN) trained independently on three sentiment- classification datasets (Yelp, IMDB, Stanford Sentiment Treebank). Using Fixed-point dynamics analysis, they locate ~500 approximate fixed points per trained network, show via PCA and locally-linear-embedding parameterization that these lie along an approximately 1D curve aligned with the classifier’s own readout direction, and confirm all three defining properties quantitatively: state-vector velocity from random initial conditions rapidly approaches zero (attracting); the top eigenvalue clusters near $|\lambda_1|\approx1$ with a time constant on the order of hundreds to thousands of tokens, while lower modes decay in tens of tokens (marginal stability, selective long-range memory); and the overlap $\mathbf{r}_1^\top\mathbf{m}$ between the top right eigenvector and the fixed-point manifold’s principal direction is far above a random-unit-vector null distribution across all fixed points (alignment). Positive/negative-valence words show large-magnitude, opposite-sign projections onto the top left eigenvector while neutral words project near zero, and PCA on hidden states shows training itself compresses a theoretically high-dimensional state space onto a low-dimensional subspace (top 2-3 PCs explain ~90% of variance, vs. much higher dimensionality in an untrained network of the same architecture). See sentiment-rnns-implement-line-attractor-dynamics.

A direct follow-up, Maheswaranathan & Sussillo (ICML 2020), shows the attractor is not the whole story: negation and intensification words (“not bad,” “extremely awesome”) deflect the state into a separate, 2-dimensional subspace orthogonal to the attractor itself, rather than moving further along it — see modifier-subspace-orthogonal-to-line-attractor-implements-negation-and-intensification for the causally-validated (subspace-ablation) details.

Haputhanthri, Storan, Jiang, Raheja, Shai, Akengin, Miolane, Schnitzer, Dinc & Tanaka (2025) — a direct extension of the Ghost Point (saddle-node bifurcation remnant) finding by an overlapping author team — show a real piecewise-linear memory RNN can form a line attractor of slow (near-zero-speed) points via a “geometric restructuring” event that occurs with no bifurcation at all, generalizing the bifurcation-driven ghost-point mechanism to a broader phase-space phenomenon. A Temporal Consistency Regularizer (penalizing frame-to-frame change in a subset of “memory” neurons) causally accelerates this attractor’s formation and enables training of strongly-connected recurrent regimes that otherwise fail — a training-time, rather than inference-time activation-patching, causal lever over the same measured geometric restructuring. See a-real-piecewise-linear-memory-rnn-forms-a-line-attractor-of-slow-points-via-geometric-restructuring-without-any-bifurcation-and-a-temporal-consistency-regularizer-causally-controls-how-readily-this-attractor-forms.

A companion, discrete phenomenon: point-attractor basins for memorized facts

Where the above evidence describes a continuous 1D manifold of fixed points used to integrate scalar evidence, Liang, Miikkulainen & Fiete (2026) locate a geometrically distinct discrete companion in the same transformer-memory setting: each fact a language model has memorized into its weights (parametric memory) corresponds to its own isolated point attractor basin in hidden-state space, not a shared continuous line. Using a controlled synthetic task (entity identifiers mapped to unique codes, installed via targeted LoRA adapters) in Qwen2.5-3B-Instruct, they show two behaviorally distinct failure modes share one geometric account: conflict (parametric memory and in-context working memory disagree) is basin competition, where working-memory input disrupts convergence to the correct basin without necessarily raising output entropy; confident hallucination (a queried fact was never learned) is convergence to the wrong or a shallow basin. A geometric margin — hidden state distance to the nearest memorized basin — separates correct recall from hallucination far more cleanly than output entropy does, and the same attractor-basin account is confirmed on natural-language factual queries from the pretrained model with no fine-tuning at all (not just the synthetic LoRA setup), and validated across a 12-model, 0.36B-14B-parameter scaling sweep. This is a discrete, per-fact point attractor structure — not the continuous 1D evidence-accumulation manifold this node otherwise describes — but shares the same fixed-point-dynamics framing: memorized facts are attracting fixed points of the network’s own recurrence-like autoregressive update, and their basin geometry, not just their existence, causally explains when recall succeeds or confidently fails. See memorized-facts-form-point-attractor-basins-in-transformer-hidden-state-space-and-basin-geometry-explains-conflict-and-hallucination.

A biological-navigation instance of the same discrete-basin companion phenomenon predates the transformer-memory case above. Jude & Hennig (2020) train a 380-unit LSTM on a memory-dependent T-maze task (sensory prediction plus Q-learning for a delayed-cue reward), with no explicit position or movement input given to the network, and find place-field mapping (30%-of-peak activity threshold) identifies well-isolated place fields collectively tiling the whole maze — a discrete point-attractor-like landscape directly analogous to biological hippocampal place cells — plus substantial extrafield firing concentrated at decision points (56% of units after reward training). A UMAP visualization of the population trajectory further shows this landscape is not static: when the agent pauses at a choice point, the representation’s manifold path abruptly switches from one trajectory’s path to the other’s before settling, mirroring non-local “sweep” activity recorded in rodent hippocampus. No causal ablation of the geometric structure itself is performed. See jude-hennig-2020-a-380-unit-lstm-trained-on-a-t-maze-forms-well-isolated-place-fields-covering-the-whole-maze-plus-extrafield-firing-at-decision-points-and-umap-shows-the-population-trajectory-abruptly-switching-manifold-paths.

Foundational precedent: fixed-point finding across three tasks

Sussillo & Barak (2013) introduce the fixed-point-finding-plus- linearization methodology this node’s evidence relies on (Fixed-point dynamics analysis), applying it to real trained (continuous-time) RNNs across three distinct tasks: a 3-bit flip-flop memory task, where it recovers exactly 8 discrete, isolated stable fixed points (a companion discrete phenomenon, not a continuous line); an input-driven sine-wave generator, where it recovers limit-cycle structure; and an integration task, where it recovers approximate marginal-stability dynamics closer to this node’s continuous-line picture. This is the founding demonstration that fixed-point topology plus local linearization can reverse-engineer a trained RNN’s own computational mechanism, predating and methodologically underlying the later continuous-line-attractor finding in sentiment RNNs. See sussillo-barak-2013-fixed-point-finding-plus-jacobian-linearization-reverse-engineers-trained-rnn-dynamics-across-three-tasks-recovering-discrete-memory-states-a-limit-cycle-and-integrator-structure.