Technical Report: Impact of Position Bias on Language Models in Token Classification

arXiv:2304.13567v3 Announce Type: replace
Abstract: Language Models (LMs) have shown state-of-the-art performance in Natural Language Processing (NLP) tasks. Downstream tasks such as Named Entity Recognition (NER) or Part-of-Speech (POS) tagging are known to suffer from data imbalance issues, particularly regarding the ratio of positive to negative examples and class disparities. This paper investigates an often-overlooked issue of encoder models, specifically the position bias of positive examples in token classification tasks. For completeness, we also include decoders in the evaluation. We …

abstract art arxiv bias classification cs.ai cs.cl data examples impact language language models language processing lms natural natural language natural language processing negative ner nlp part part-of-speech performance positive processing recognition report speech state tagging tasks technical token type