Phoneme identity is linearly decodable from a visually grounded speech RNN, most strongly in lower recurrent layers

- A pre-trained Recurrent Highway Network (the Speech COCO model of Chrupała et al. 2017a, 5 recurrent layers, trained on synthetically spoken COCO captions to map speech and images into a joint space) encodes phoneme identity in its hidden activations: L2-penalized logistic-regression probes recover phonemes with error far below the 89% majority baseline, markedly better than from MFCC input. [alishahi-etal-2017-phonology-rnn-grounded-speech] - Phoneme decodability is highest at recurrent layers 1 and 2 and decreases at higher layers, though even the last recurrent layer still encodes phoneme identity to a substantial degree; the attention layer producing utterance embeddings attenuates phonological information. [alishahi-etal-2017-phonology-rnn-grounded-speech] - The model is trained with weak supervision from the visual modality rather than phonemic labels; the analyzed model is a single small trained network (~9.8M parameters) on synthetic speech, so this is a directional result about this model only. [alishahi-etal-2017-phonology-rnn-grounded-speech]