Validating and Exploring Large Geographic Corpora

arXiv:2403.08198v1 Announce Type: new
Abstract: This paper investigates the impact of corpus creation decisions on large multi-lingual geographic web corpora. Beginning with a 427 billion word corpus derived from the Common Crawl, three methods are used to improve the quality of sub-corpora representing specific language-country pairs like New Zealand English: (i) the agreement of independent language identification systems, (ii) hash-based deduplication, and (iii) location-specific outlier detection. The impact of each of these steps is then evaluated at the language level …

abstract agreement arxiv billion country cs.cl decisions english impact language new zealand paper quality type web word