Discovering Latent Knowledge in Language Models Without Supervision

arXiv:2212.03827v2 Announce Type: replace-cross
Abstract: Existing techniques for training language models can be misaligned with the truth: if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can't detect. We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. Specifically, we introduce a method for …

abstract arxiv cs.ai cs.cl cs.lg errors generate human humans imitation learning knowledge language language models rate supervision text them train training truth type