Multi-Word Tokenization for Sequence Compression

arXiv:2402.09949v1 Announce Type: cross
Abstract: Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this pa005 per, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a …

abstract arxiv beyond compression computational cost cs.cl cs.lg industrial language language models large language large language models modelling per tasks tokenization tokens type word