Ghost points of destroyed saddle-nodes govern abrupt RNN learning
measured in 1 paper- A ghost point - a near-zero (not exactly zero) minimum of the speed function, the residual of a fixed point destroyed by a saddle-node bifurcation - creates a transient "no-learning zone" that produces long plateaus and then abrupt, grokking-like jumps in learning. [dinc-etal-2025-ghost-mechanism] - Reducing the dynamics near the ghost to a 1-D canonical form, the paper derives (Eq. 15) a critical learning rate alpha* = (3 pi^4 / 32) T^-5 (scaling as T^-5), with the optimal scale parameter r* = pi^2/(4 T^2) scaling as T^-2. [dinc-etal-2025-ghost-mechanism] - Confirmed in gradient-trained vanilla RNNs (N=100; rank-one primary, plus a rank sweep to full rank) on a delayed-activation working-memory task; the critical learning rate to escape the no-learning zone grows with trainable rank. [dinc-etal-2025-ghost-mechanism] - The authors caution the exact T^-5 law is a toy-model result not expected to transfer quantitatively to full-rank RNNs. [dinc-etal-2025-ghost-mechanism]