Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding

arXiv:2301.03765v2 Announce Type: replace-cross
Abstract: Current natural language understanding (NLU) models have been continuously scaling up, both in terms of model size and input context, introducing more hidden and input neurons. While this generally improves performance on average, the extra neurons do not yield a consistent improvement for all instances. This is because some hidden neurons are redundant, and the noise mixed in input neurons tends to distract the model. Previous work mainly focuses on extrinsically reducing low-utility neurons by …

abstract arxiv consistent context cs.cl cs.ir cs.lg current extra hidden improvement instances language language understanding loss natural natural language neurons nlu performance scaling scaling up terms type understanding utility