Trained vanilla ReLU RNNs emulating hidden Markov models collapse to a single fixed point without input and show noise-sustained rotation along closed orbits whose phase modulates emission probabilities

- Torre, Viscione, Pompe, Grewe and Mante train vanilla ReLU RNNs (50-200 hidden units, Gaussian input) with a Sinkhorn objective to replicate HMM emission statistics, and reverse-engineer them: without input activity collapses to a single fixed point, whereas under stochastic input trajectories follow noise-sustained dynamics along closed orbits whose rotation modulates emission probabilities, with transitions between slow and fast regions initiated by a few kick neurons and the same motif reused across fully connected, cyclic and linear-chain HMMs. The orbit is identified in projections onto the first two or three principal components; its radius scales linearly with input variance, and for cyclic and fully connected HMMs the RNNs develop multiple orbits connecting distinct pairs of slow regions rather than a single loop. Because without input the activity collapses to a single fixed point, the closed orbit is noise-sustained and is not a limit cycle of the deterministic dynamics; it is filed under manifolds-circle as a closed one-dimensional orbit (topology assessed only from low-dimensional PCA projections), a choice flagged for review against limit-cycle. [torre-etal-2025-mechanistic-interpretability-of-rnns-emulating-hidden-markov-models]