Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement

arXiv:2403.13754v1 Announce Type: new
Abstract: The relationship between language model tokenization and performance is an open area of research. Here, we investigate how different tokenization schemes impact number agreement in Spanish plurals. We find that morphologically-aligned tokenization performs similarly to other tokenization schemes, even when induced artificially for words that would not be tokenized that way during training. We then present exploratory analyses demonstrating that language model embeddings for different plural tokenizations have similar distributions along the embedding space axis …

abstract agreement arxiv cs.cl impact language language model performance relationship research spanish tokenization type words