Tokenization Preference for Human and ML Model: An Annotation Study

arXiv:2304.10813v2 Announce Type: replace
Abstract: Is preferred tokenization for humans also preferred for machine-learning (ML) models? This study examines the relations between preferred tokenization for humans (appropriateness and readability) and one for ML models (performance on an NLP task). The question texts of the Japanese commonsense question-answering dataset are tokenized with six different tokenizers, and the performances of human annotators and ML models were compared. Furthermore, we analyze relations among performance of answers by human and ML model, the appropriateness …

abstract annotation arxiv cs.cl dataset human humans japanese machine ml models nlp performance question readability relations study tokenization type