Do Llamas Work in English? On the Latent Language of Multilingual Transformers

arXiv:2402.10588v1 Announce Type: new
Abstract: We ask whether multilingual language models trained on unbalanced, English-dominated corpora use English as an internal pivot language -- a question of key importance for understanding how language models function and the origins of linguistic bias. Focusing on the Llama-2 family of transformer models, our study uses carefully constructed non-English prompts with a unique correct single-token continuation. From layer to layer, transformers gradually map an input embedding of the final prompt token to an output …

abstract arxiv bias cs.cl cs.cy english family function importance key language language models llama llamas multilingual pivot question transformer transformers type understanding work