Web: http://arxiv.org/abs/2204.01171

May 11, 2022, 1:12 a.m. | Kushal Arora, Layla El Asri, Hareesh Bahuleyan, Jackie Chi Kit Cheung

cs.LG updates on arXiv.org arxiv.org

Current language generation models suffer from issues such as repetition,
incoherence, and hallucinations. An often-repeated hypothesis is that this
brittleness of generation models is caused by the training and the generation
procedure mismatch, also referred to as exposure bias. In this paper, we verify
this hypothesis by analyzing exposure bias from an imitation learning
perspective. We show that exposure bias leads to an accumulation of errors,
analyze why perplexity fails to capture this accumulation, and empirically show
that this accumulation …

arxiv bias imitation learning language language generation learning perspective

More from arxiv.org / cs.LG updates on arXiv.org

Data Analyst, Patagonia Action Works

@ Patagonia | Remote

Data & Insights Strategy & Innovation General Manager

@ Chevron Services Company, a division of Chevron U.S.A Inc. | Houston, TX

Faculty members in Research areas such as Bayesian and Spatial Statistics; Data Privacy and Security; AI/ML; NLP; Image and Video Data Analysis

@ Ahmedabad University | Ahmedabad, India

Director, Applied Mathematics & Computational Research Division

@ Lawrence Berkeley National Lab | Berkeley, Ca

Business Data Analyst

@ MainStreet Family Care | Birmingham, AL

Assistant/Associate Professor of the Practice in Business Analytics

@ Georgetown University McDonough School of Business | Washington DC