Web: http://arxiv.org/abs/2205.02517

May 6, 2022, 1:11 a.m. | Shaojie Jiang, Ruqing Zhang, Svitlana Vakulenko, Maarten de Rijke

cs.CL updates on arXiv.org arxiv.org

The cross-entropy objective has proved to be an all-purpose training
objective for autoregressive language models (LMs). However, without
considering the penalization of problematic tokens, LMs trained using
cross-entropy exhibit text degeneration. To address this, unlikelihood training
has been proposed to force unlikely tokens to be assigned a low probability by
a LM. But unlikelihood does not consider the relationship between the label
tokens and the unlikely token candidates, thus showing marginal improvements in
degeneration. We propose a new contrastive token …

arxiv learning neural text

Data Analyst, Patagonia Action Works

@ Patagonia | Remote

Data & Insights Strategy & Innovation General Manager

@ Chevron Services Company, a division of Chevron U.S.A Inc. | Houston, TX

Faculty members in Research areas such as Bayesian and Spatial Statistics; Data Privacy and Security; AI/ML; NLP; Image and Video Data Analysis

@ Ahmedabad University | Ahmedabad, India

Director, Applied Mathematics & Computational Research Division

@ Lawrence Berkeley National Lab | Berkeley, Ca

Business Data Analyst

@ MainStreet Family Care | Birmingham, AL

Assistant/Associate Professor of the Practice in Business Analytics

@ Georgetown University McDonough School of Business | Washington DC