Ghost Point (saddle-node bifurcation remnant)

structures / Manifolds / Ghost Point (saddle-node bifurcation remnant)

Ghost Point (saddle-node bifurcation remnant)

OBJECTdynamical-objectadvancedhow it's classified →

A transient dynamical bottleneck in a recurrent network's state space — not itself a fixed point, but a remnant of one that vanished via a saddle-node bifurcation earlier in training — identified via a near-zero eigenvalue of the network's speed-function Jacobian, and shown to slow learning by trapping the trajectory during training.

Replicationcomputed from the corpus — never hand-assigned

1 paper1 architecture class1 domain1 model family

Filled = two or more values reported by papers that share no author — replication. Outlined = two or more values, but all from a single study — breadth, not replication. Grey = a single value. Derived from paper authorship and each model's architecture class, domain and family; it updates itself when a paper is added.

Definition

For a recurrent network with update rule $h_{t+1}=F(h_t,x_t)$ , define a speed/energy function $s_\theta(h) = \tfrac12\|h - F(h,0)\|^2$ whose local minima correspond to (near-)fixed points of the autonomous dynamics. A ghost point is a point where $s_\theta$ has a very small but nonzero local minimum — the Jacobian of the dynamics has one eigenvalue very close to zero (rather than exactly zero, as at a true fixed point) and the rest negative — arising as the residual trace of a fixed point that existed earlier in training and disappeared via a saddle-node bifurcation as parameters changed. Unlike Line Attractor (a whole 1-manifold of marginally-stable true fixed points, exploited functionally for evidence integration), a ghost point is a single, transient, imperfect bottleneck that slows — rather than stabilizes — trajectories passing near it.

Relative to line-attractor

A Line Attractor is functional machinery: a manifold of genuine fixed points a trained network exploits to integrate evidence over time, and remains present at convergence. A ghost point is instead diagnostic of a training-dynamics obstruction: it is what remains after a fixed point has been destroyed by a bifurcation, and its presence — not its function — is what matters, since trajectories linger near it and this lingering is mechanistically responsible for plateaus in the learning curve (slow, then abrupt, “grokking-like” jumps in task performance).

Key evidence

Dinc, Cirakman, Kurtkaya, Yuksekgonul, Jiang, Schnitzer & Tanaka (2025) analytically derive and empirically confirm ghost-point dynamics in real trained rank-one, rank-two, and full-rank vanilla RNNs (N=100 neurons) on a delayed-activation working-memory task, tracking specific ghost-point trajectories via PCA-projected state space across training epochs and deriving (Eq. 15) a critical-learning-rate scaling $\alpha^*=(3\pi^4/32)\,T^{-5}$ (i.e. $\propto T^{-5}$ ) for the canonical rank-one model, with the optimal scale parameter $r^*=\pi^2/(4T^2)\propto T^{-2}$ ; the critical learning rate needed to escape the ghost-induced “no-learning zone” grows with trainable rank, though the authors caution the exact $T^{-5}$ law is a toy-model result not expected to transfer quantitatively to full-rank RNNs. See dinc-etal-2025-ghost-points-remnants-of-destroyed-saddle-node-fixed-points-govern-abrupt-learning-transitions-in-real-trained-rnns.

Key papers

Dinc, Fatih; Cirakman, Ege; Kurtkaya, Bariscan; Yuksekgonul, Mert; Jiang, Yiqi; Schnitzer, Mark J.; Tanaka, Hidenori (2025). A Ghost Mechanism: An Analytical Model of Abrupt Learning in Recurrent Networks. Physical Review X, arXiv:2501.02378.

Found in (1 observation · 1 family)

Vanilla tanh RNN (task-trained, N=100 neurons)

A Ghost Mechanism: An Analytical Model of Abrupt Learning in Recurrent Networks (2025)measured

Ghost points of destroyed saddle-nodes govern abrupt RNN learning

Details

- A ghost point - a near-zero (not exactly zero) minimum of the speed function, the residual of a fixed point destroyed by a saddle-node bifurcation - creates a transient "no-learning zone" that produces long plateaus and then abrupt, grokking-like jumps in learning. [dinc-etal-2025-ghost-mechanism] - Reducing the dynamics near the ghost to a 1-D canonical form, the paper derives (Eq. 15) a critical learning rate alpha* = (3 pi^4 / 32) T^-5 (scaling as T^-5), with the optimal scale parameter r* = pi^2/(4 T^2) scaling as T^-2. [dinc-etal-2025-ghost-mechanism] - Confirmed in gradient-trained vanilla RNNs (N=100; rank-one primary, plus a rank sweep to full rank) on a delayed-activation working-memory task; the critical learning rate to escape the no-learning zone grows with trainable rank. [dinc-etal-2025-ghost-mechanism] - The authors caution the exact T^-5 law is a toy-model result not expected to transfer quantitatively to full-rank RNNs. [dinc-etal-2025-ghost-mechanism]

models: Rank-one RNN (N=100, working-memory tasks, Dinc et al. 2025), Full-rank RNN (N=100, working-memory tasks, Dinc et al. 2025) · method: Fixed-point dynamics analysis