Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training

arXiv:2404.05560v1 Announce Type: new
Abstract: Chinese sequence labeling tasks are heavily reliant on accurate word boundary demarcation. Although current pre-trained language models (PLMs) have achieved substantial gains on these tasks, they rarely explicitly incorporate boundary information into the modeling process. An exception to this is BABERT, which incorporates unsupervised statistical boundary information into Chinese BERT's pre-training objectives. Building upon this approach, we input supervised high-quality boundary information to enhance BABERT's learning, developing a semi-supervised boundary-aware PLM. To assess PLMs' ability …

abstract arxiv chinese cs.cl current exception information labeling language language model language models modeling pre-training process semi-supervised statistical tasks training type unsupervised word