How Important Is Tokenization in French Medical Masked Language Models?

arXiv:2402.15010v1 Announce Type: cross
Abstract: Subword tokenization has become the prevailing standard in the field of natural language processing (NLP) over recent years, primarily due to the widespread utilization of pre-trained language models. This shift began with Byte-Pair Encoding (BPE) and was later followed by the adoption of SentencePiece and WordPiece. While subword tokenization consistently outperforms character and word-level tokenization, the precise factors contributing to its success remain unclear. Key aspects such as the optimal segmentation granularity for diverse tasks …

abstract adoption arxiv become began cs.ai cs.cl cs.lg encoding french language language models language processing medical natural natural language natural language processing nlp processing shift standard tokenization type