ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media Text

Lexical normalization, a fundamental task in Natural Language Processing (NLP), involves the transformation of words into their canonical forms. This process has been proven to benefit various downstream NLP tasks greatly. In this work, we introduce Vietnamese Lexical Normalization (ViLexNorm), the first-ever corpus developed for the Vietnamese lexical normalization task. The corpus comprises over 10,000 pairs of sentences meticulously annotated by human annotators, sourced from public comments on Vietnam's most popular social media platforms. Various methods were used to evaluate …

benefit canonical cs.cl forms language language processing media natural natural language natural language processing nlp normalization process processing social social media tasks text transformation words work