Semiparametric Token-Sequence Co-Supervision

arXiv:2403.09024v1 Announce Type: new
Abstract: In this work, we introduce a semiparametric token-sequence co-supervision training method. It trains a language model by simultaneously leveraging supervision from the traditional next token prediction loss which is calculated over the parametric token embedding space and the next sequence prediction loss which is calculated over the nonparametric sequence embedding space. The nonparametric sequence embedding space is constructed by a separate language model tasked to condense an input text into a single representative embedding. Our …

abstract arxiv cs.ai cs.cl embedding language language model loss next parametric prediction space supervision token training trains type work