Fixed-point graph topology (population-level universality of RNN dynamical structure)

structures / Manifolds / Fixed-point graph topology (population-level universality of RNN dynamical structure)

Fixed-point graph topology (population-level universality of RNN dynamical structure)

OBJECTdynamical-objectadvancedhow it's classified →

The combinatorial-plus-geometric structure formed by a recurrent network's full set of fixed and slow points together with the heteroclinic orbits connecting them — a graph-like dynamical object, distinct from a single line-attractor's 1-manifold of marginally-stable points, that is found to recur with quantitatively similar topology across large, independently-trained populations of RNNs solving the same task, regardless of architecture (vanilla, UGRNN, GRU, LSTM).

Replicationcomputed from the corpus — never hand-assigned

3 papers · no shared authors1 architecture class2 domains · across papers3 model families · across papers

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

For a trained RNN with update rule $h_{t+1}=F(h_t,x_t)$ , locate the full set of fixed and slow points of the autonomous ( $x=0$ ) dynamics via numerical optimization of the speed function $s(h)=\|h-F(h,0)\|^2$ , then linearize around each to obtain its stability type (stable node, saddle, etc.) and connect points via the heteroclinic orbits the dynamics actually traverse between them. The resulting object is a graph: a discrete vertex set (fixed/slow points, each with a local stability type) plus edges (heteroclinic connections realized by trajectories), embedded within the low-dimensional manifold the state trajectories occupy. This is a richer combinatorial-plus-geometric description than a single continuous attractor manifold — it is used to characterize how many qualitatively distinct dynamical regimes a network has learned and how they are wired together, not just the shape of one of them.

Relative to line-attractor and ghost-point

Line Attractor is a single, specific fixed-point-graph instance: one connected 1-manifold of marginally-stable fixed points, used for scalar evidence integration. Ghost Point (saddle-node bifurcation remnant) is a single transient, imperfect fixed point left behind after a bifurcation, diagnostic of a training obstruction rather than final task-solving structure. This node is the more general object one level up: the entire population of fixed points (of every stability type — attracting, saddle, or otherwise) a converged network has, plus the heteroclinic connections between them — of which a line-attractor or a small ensemble of ghost points can be special cases or sub-components. Critically, this node’s defining evidence is about topological universality: whether independently optimized networks, even of different architectures, converge on graphs with the same qualitative wiring.

Key evidence

Maheswaranathan, Williams, Golub, Ganguli & Sussillo (2019) train, for each of three tasks (3-bit flip-flop, frequency-cued sine wave, context-dependent integration), 2,880 RNNs (four architectures — vanilla RNN, UGRNN, GRU, LSTM — x relu/tanh x three sizes x four L2 strengths x 30 seeds), then locate each network’s fixed/slow points numerically (with static inputs specifying the task condition), linearize them and build a fixed-point graph from heteroclinic transitions. They report that the fixed-point topology, limit cycles and linearized dynamics “often appear universal across all architectures”, while representational geometry measured with SVCCA/CKA clusters by architecture and activation function. The universality is not uniform: for sine-wave generation the topology embedding showed no structure by architecture or activation, and for context-dependent integration the line-attractor topology clustered by activation function but not by architecture; the fixed-point graph comparison discards networks whose number of fixed points differs from the modal count (under 10% of the flip-flop population). See maheswaranathan-etal-2019-fixed-point-graph-topology-is-universal-across-2880-independently-trained-rnns-of-four-architectures-on-the-same-task.

Kononov, Pospelov, Anokhin, Nekorkin & Maslennikov (2025) analyze 250-unit RNNs trained from scratch on a context-dependent decision-making task with reinforcement learning (PPO) versus supervised learning (Adam plus cross-entropy), using numerical fixed-point finding. RL-trained networks show hybrid dynamics combining stable fixed points with quasi-periodic attractors, whereas SL-trained networks converge almost exclusively to fixed-point-only landscapes. See kononov-etal-2025-rl-trained-rnns-spontaneously-discover-hybrid-fixed-point-plus-quasi-periodic-attractor-architectures.

Ren & Liu (2026) mechanistically analyze a real Hierarchical Reasoning Model (HRM), a hierarchical recurrent architecture trained from scratch on Sudoku-Extreme, finding it can fail on trivially simple puzzles (e.g. only one unknown cell), tracing this to a violation of HRM’s own fixed-point-convergence assumption: rather than a single fixed point, the model’s recurrent dynamics possess multiple fixed points, and HRM converges to whichever one its recurrent iteration reaches first — which can be an incorrect answer even when the correct answer is trivial to determine. This “guessing, not reasoning” dynamical signature (distinct from this node’s usual population-level topology-universality claims, here a single-network multiple-fixed-point pathology) co-occurs with a grokking-like non-uniform improvement in answer correctness tied to specific critical reasoning steps; diagnosing this motivates data-augmentation, input-perturbation, and model-bootstrapping fixes (“Augmented HRM”) that raise Sudoku-Extreme accuracy from 55.0% to 96.9%. See a-hierarchical-reasoning-model-trained-on-sudoku-extreme-develops-multiple-fixed-points-in-its-recurrent-dynamics-and-guesses-among-them-rather-than-reasoning.

Key papers

Maheswaranathan, N., Williams, A., Golub, M., Ganguli, S. & Sussillo, D. (2019). Universality and Individuality in Neural Dynamics Across Large Populations of Recurrent Networks. NeurIPS 2019, arXiv:1907.08549 — origin of the population-scale fixed-point-graph universality finding.

Found in (3 observations · 3 families)

Hierarchical Reasoning Model (HRM)

Are Your Reasoning Models Reasoning or Guessing? A Mechanistic Analysis of Hierarchical Reasoning Models (2026)measured

A real trained Hierarchical Reasoning Model's recurrent dynamics develop multiple fixed points rather than a single one, and the model 'guesses' by converging to whichever fixed point it reaches first -- which can be wrong even on trivially simple puzzles

Details

- Ren & Liu mechanistically analyze a real Hierarchical Reasoning Model (HRM) trained from scratch on Sudoku-Extreme, finding (largely from qualitative PCA-projected trajectories) it can fail on extremely simple puzzles (e.g. those with only one unknown cell), tracing this to violation of the fixed-point property that is a fundamental assumption of HRM's recurrent-convergence design. [ren-liu-2026-reasoning-or-guessing-hrm] - The model's recurrent dynamics are found to possess multiple fixed points rather than a single one; HRM converges to whichever fixed point its recurrent iteration reaches first, and this first-reached fixed point can be an incorrect answer even when the correct answer is trivial to determine -- a "guessing," rather than "reasoning," dynamical signature. [ren-liu-2026-reasoning-or-guessing-hrm] - This multiple-fixed-point structure co-occurs with a "grokking"-like training dynamic in which answer correctness improves non-uniformly, tied to specific critical reasoning steps; based on this diagnosis, the authors propose data augmentation, input perturbation, and model-bootstrapping strategies (an "Augmented HRM") that raise Sudoku-Extreme accuracy to 96.9% for the Augmented HRM, versus 55.0% for the baseline HRM. [ren-liu-2026-reasoning-or-guessing-hrm]

models: Hierarchical Reasoning Model (~27M, trained from scratch per task, e.g. Sudoku-Extreme) · method:

Kononov et al. 2025 Context-Dependent Decision-Making RNN

Reinforcement learning gives rise to emergent hybrid computational dynamics in recurrent neural networks (2025)measured

RL-trained RNNs show hybrid fixed-point plus quasi-periodic attractor dynamics, whereas SL-trained RNNs on the same task converge almost exclusively to fixed-point-only solutions

Details

- Kononov, Pospelov, Anokhin, Nekorkin & Maslennikov train 250-unit vanilla RNNs from scratch on a context-dependent decision-making (CtxDM) task using either reinforcement learning (PPO) or supervised learning (Adam + cross-entropy) on the identical task, then analyze each trained network's dynamics by numerical fixed-point finding. [kononov-etal-2025-hybrid-computational-dynamics-rl] - RL-trained RNNs show hybrid dynamics: "stable fixed points, where neural activity converges to a single static pattern, and quasi-periodic attractors" (Results), whereas SL-trained RNNs on the same task converge almost exclusively to fixed-point-only landscapes. The quasi-periodic attractors are reported in the paper's terms; mapping them to a limit-cycle node is not asserted here. [kononov-etal-2025-hybrid-computational-dynamics-rl]

models: Kononov et al. 2025 CtxDM ReLU-RNN (250 units, RL- and SL-trained) · method:

Maheswaranathan et al. 2019 RNN population study (Vanilla/UGRNN/GRU/LSTM)

Universality and Individuality in Neural Dynamics Across Large Populations of Recurrent Networks (2019)measured

Fixed-point graph topology is universal across large populations of independently-trained RNNs

Details

- For each of three tasks (3-bit flip-flop, frequency-cued sine wave, context-dependent integration) the authors train 2,880 RNNs (4 architectures -- vanilla, UGRNN, GRU, LSTM -- x relu/tanh x 3 sizes x 4 L2 strengths x 30 seeds; LSTM-relu did not train for the sine-wave task). They find that the fixed-point topology (fixed points, their stability, heteroclinic transitions, limit cycles, linearized dynamics) "often appears universal across all architectures", while representational geometry (SVCCA/CKA) clusters by architecture and activation function. [maheswaranathan-etal-2019-universality-individuality-rnn-populations] - Fixed points are located numerically (with static inputs specifying the task condition), linearized to determine stability, and converted into a directed graph via heteroclinic transition probabilities; in the flip-flop task, networks with a number of fixed points different from the modal count (less than 10% of the population) were discarded before comparing graphs. [maheswaranathan-etal-2019-universality-individuality-rnn-populations] - The universality is task-dependent in detail: for sine-wave generation the topology embedding showed no structure varying with architecture or activation, while for context-dependent integration the line-attractor topology did not cluster by architecture but did cluster by activation function; linearized integration modes differ (gated RNNs use more eigenvalues near 1 than vanilla RNNs). [maheswaranathan-etal-2019-universality-individuality-rnn-populations]

models: Population of ~2,880 independently-trained RNNs (Vanilla/UGRNN/GRU/LSTM) · method: Fixed-point dynamics analysis, Singular Vector Canonical Correlation Analysis (SVCCA), Centered Kernel Alignment (CKA)