Cross-Lingual Representation Analysis

How do multilingual transformer models represent similar meanings across different languages — and do they do it the same way across all layers?

Large language models trained on multiple languages can perform cross-lingual tasks, but the mechanisms behind this ability are not fully understood.

This research investigates whether multilingual representations capture shared semantic structures across languages and how these representations change throughout different model layers.

Understanding these internal representations can help researchers better understand multilingual model behavior and improve future language model architectures.

Multilingual transformer models such as mBERT are trained on text from many languages and can transfer knowledge between languages despite not receiving explicit language alignment instructions.

Previous research has shown that multilingual models develop shared representation spaces, but important questions remain:

• How aligned are representations across languages?
• Which model layers contain stronger cross-lingual information?
• How does semantic similarity evolve throughout the network?

This project explores these questions by analyzing the internal representations learned by multilingual transformers.

The research pipeline involved extracting and analyzing hidden representations from mBERT across multiple layers.

The methodology included:

Representation Extraction

Extracted hidden-state embeddings from different transformer layers using mBERT (13 total hidden states: embedding layer + 12 transformer layers).

Word-Level Alignment

Used identical English and Spanish audiobook transcripts as stimuli, with 404 perfectly aligned translation pairs from the MUSE dataset.

Representation Analysis

Applied techniques including:

• Representational Similarity Analysis (RSA)
• Similarity measurements between embedding spaces
• Dimensionality analysis
• Statistical evaluation

Cross-Language Comparison

Compared how semantic structures are preserved across languages throughout the model.

The analysis revealed a two-level dissociation in mBERT that mirrors findings from human hippocampal recordings of bilingual speakers:

1. Individual dimensions are language-specific. Across all 13 layers, the cross-language Pearson correlation of all 768 dimensions shows that individual features diverge between languages.

2. Population geometry is language-independent. While individual dimensions diverge, the global semantic manifold is shared across languages — the geometric structure of meaning is preserved.

The project also uncovered a novel paradox: representational similarity (RSA) peaks at layer 0 (rho = 0.227) while true translation similarity peaks deeper in the network, suggesting that surface-level form alignment and deep semantic alignment are computationally distinct.

The project was implemented using Python-based deep learning tools.

Model

mBERT (bert-base-multilingual-cased) — Used to extract hidden representations across transformer layers.

Frameworks

PyTorch — Used for model execution and tensor operations.

Hugging Face Transformers — Used for loading and interacting with pretrained transformer models.

Data Processing

NumPy + Pandas — Used for numerical processing, dataset management, and analysis pipelines.

Visualization

Matplotlib — Used to visualize representation patterns and research findings.

Analyzing internal model representations provides insight into how multilingual models organize knowledge.

Understanding these structures can help improve:

• Multilingual model design
• Cross-language transfer
• Model interpretability
• Evaluation methods for language models

The project also highlights that studying representations can reveal behaviors that are not obvious from model outputs alone.

The next step is extending this methodology beyond encoder-based models toward decoder-based language models such as Llama-based architectures.

Future directions include:

• Studying cross-lingual representations in large decoder models
• Comparing representation evolution across architectures
• Investigating how multilingual capabilities emerge during training
• Exploring methods for improving multilingual alignment