RNNLM (Recurrent Neural Network Language Model)

Independent / academic (Mikolov et al.)

Structures found in this family (2)

Linear Direction, Representational-similarity trajectory across depth

By model (2)

RNNLM word vectors (Mikolov, Yih & Zweig 2013, Broadcast News)

Linear Direction

LSTM language model (AWD-LSTM setup, PTB/WikiText-2)

Representational-similarity trajectory across depth

Observations (2)

The foundational king-man+woman parallelogram analogy in RNNLM vectors

Wider networks converge to more similar representations, measured by PWCCA

Papers

Linguistic Regularities in Continuous Space Word Representations (2013), Insights on Representational Similarity in Neural Networks with Canonical Correlation Analysis (2018)