Pre-Trained Language Models Represent Some Geographic Populations Better Than Others

arXiv:2403.11025v1 Announce Type: new
Abstract: This paper measures the skew in how well two families of LLMs represent diverse geographic populations. A spatial probing task is used with geo-referenced corpora to measure the degree to which pre-trained language models from the OPT and BLOOM series represent diverse populations around the world. Results show that these models perform much better for some populations than others. In particular, populations across the US and the UK are represented quite well while those in …

abstract arxiv bloom cs.cl diverse families geo language language models llms paper series skew spatial type