Fixed-point graph topology is universal across large populations of independently-trained RNNs

- For each of three tasks (3-bit flip-flop, frequency-cued sine wave, context-dependent integration) the authors train 2,880 RNNs (4 architectures -- vanilla, UGRNN, GRU, LSTM -- x relu/tanh x 3 sizes x 4 L2 strengths x 30 seeds; LSTM-relu did not train for the sine-wave task). They find that the fixed-point topology (fixed points, their stability, heteroclinic transitions, limit cycles, linearized dynamics) "often appears universal across all architectures", while representational geometry (SVCCA/CKA) clusters by architecture and activation function. [maheswaranathan-etal-2019-universality-individuality-rnn-populations] - Fixed points are located numerically (with static inputs specifying the task condition), linearized to determine stability, and converted into a directed graph via heteroclinic transition probabilities; in the flip-flop task, networks with a number of fixed points different from the modal count (less than 10% of the population) were discarded before comparing graphs. [maheswaranathan-etal-2019-universality-individuality-rnn-populations] - The universality is task-dependent in detail: for sine-wave generation the topology embedding showed no structure varying with architecture or activation, while for context-dependent integration the line-attractor topology did not cluster by architecture but did cluster by activation function; linearized integration modes differ (gated RNNs use more eigenvalues near 1 than vanilla RNNs). [maheswaranathan-etal-2019-universality-individuality-rnn-populations]