Methods for Estimating and Improving Robustness of Language Models. (arXiv:2206.08446v1 [cs.CL])

Web: http://arxiv.org/abs/2206.08446

Despite their outstanding performance, large language models (LLMs) suffer
notorious flaws related to their preference for simple, surface-level textual
relations over full semantic complexity of the problem. This proposal
investigates a common denominator of this problem in their weak ability to
generalise outside of the training domain. We survey diverse research
directions providing estimations of model generalisation ability and find that
incorporating some of these measures in the training objectives leads to
enhanced distributional robustness of neural models. Based on …

arxiv language language models models robustness