Aligning language models with human preferences

arXiv:2404.12150v1 Announce Type: new
Abstract: Language models (LMs) trained on vast quantities of text data can acquire sophisticated skills such as generating summaries, answering questions or generating code. However, they also manifest behaviors that violate human preferences, e.g., they can generate offensive content, falsehoods or perpetuate social biases. In this thesis, I explore several approaches to aligning LMs with human preferences. First, I argue that aligning LMs can be seen as Bayesian inference: conditioning a prior (base, pretrained LM) on …

abstract arxiv biases code cs.cl cs.lg data explore generate however human language language models lms manifest questions skills social text thesis type vast