Modeling Orthographic Variation in Occitan's Dialects

arXiv:2404.19315v1 Announce Type: new
Abstract: Effectively normalizing textual data poses a considerable challenge, especially for low-resource languages lacking standardized writing systems. In this study, we fine-tuned a multilingual model with data from several Occitan dialects and conducted a series of experiments to assess the model's representations of these dialects. For evaluation purposes, we compiled a parallel lexicon encompassing four Occitan dialects. Intrinsic evaluations of the model's embeddings revealed that surface similarity between the dialects strengthened representations. When the model was …

abstract arxiv challenge cs.cl data evaluation languages low modeling multilingual series study systems textual type variation writing