Representational Similarity Analysis (RSA) is introduced as a technique for comparing how different language models encode words internally. Rather than directly comparing embedding vectors between models like BERT and GPT-2 (which fails due to random initialization and different embedding spaces), RSA computes cosine similarities between word pairs within each model and then correlates those similarity patterns across models. The post walks through extracting embeddings from BERT-large and GPT-2-medium, demonstrating that direct correlation yields near-zero results, while RSA reveals shared semantic structure. Code is provided via GitHub/Google Colab. Part 2 will extend the analysis to transformer layers and category selectivity.
Table of contents
What you will learn in this 2-part post seriesHow to use the code with these postsWhat are “embeddings” in language models like Claude?How can we compare different embeddings?Extract embeddings from two LLMsDirectly comparing model embeddingsRepresentational similarity analysisRSA with different embeddings sizesConclusions and what’s coming up in the next post783 Impressions