Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels

arXiv:2404.14313v1 Announce Type: new
Abstract: When prompting a language model (LM), users frequently expect the model to adhere to a set of behavioral principles across diverse tasks, such as producing insightful content while avoiding harmful or biased language. Instilling such principles into a model can be resource-intensive and technically challenging, generally requiring human preference labels or examples. We introduce SAMI, a method for teaching a pretrained LM to follow behavioral principles that does not require any preference labels or demonstrations. …

abstract alignment arxiv cs.cl diverse expect information labels language language model prompting set tasks type