all AI news
The Geometry of Multilingual Language Model Representations. (arXiv:2205.10964v2 [cs.CL] UPDATED)
cs.CL updates on arXiv.org arxiv.org
We assess how multilingual language models maintain a shared multilingual
representation space while still encoding language-sensitive information in
each language. Using XLM-R as a case study, we show that languages occupy
similar linear subspaces after mean-centering, evaluated based on causal
effects on language modeling performance and direct comparisons between
subspaces for 88 languages. The subspace means differ along language-sensitive
axes that are relatively stable throughout middle layers, and these axes encode
information such as token vocabularies. Shifting representations by language …
arxiv geometry language language model multilingual language model