Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies

arXiv:2312.11779v3 Announce Type: replace-cross
Abstract: Gender-inclusive NLP research has documented the harmful limitations of gender binary-centric large language models (LLM), such as the inability to correctly use gender-diverse English neopronouns (e.g., xe, zir, fae). While data scarcity is a known culprit, the precise mechanisms through which scarcity affects this behavior remain underexplored. We discover LLM misgendering is significantly influenced by Byte-Pair Encoding (BPE) tokenization, the tokenizer powering many popular LLMs. Unlike binary pronouns, BPE overfragments neopronouns, a direct consequence of …

abstract arxiv behavior binary cs.ai cs.cl cs.lg data diverse english gender language language models large language large language models limitations llm nlp research technologies through tokenization type