Token Alignment via Character Matching for Subword Completion

arXiv:2403.08688v1 Announce Type: new
Abstract: Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens. This struggle stems from tokenization, where partial tokens fall out of distribution during inference, leading to incorrect or nonsensical outputs. This paper examines a technique to alleviate the tokenization artifact on text completion in generative models, maintaining performance even in regular non-subword cases. The method, termed token alignment, involves backtracking to the last complete tokens and ensuring the model's …

abstract alignment applications artifact arxiv cs.ai cs.cl distribution generative generative models inference paper prompts struggle token tokenization tokens type via