Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo

arXiv:2404.17546v1 Announce Type: new
Abstract: Numerous capability and safety techniques of Large Language Models (LLMs), including RLHF, automated red-teaming, prompt engineering, and infilling, can be cast as sampling from an unnormalized target distribution defined by a given reward or potential function over the full sequence. In this work, we leverage the rich toolkit of Sequential Monte Carlo (SMC) for these probabilistic inference problems. In particular, we use learned twist functions to estimate the expected future value of the potential at …

abstract arxiv automated capability cs.ai cs.cl cs.lg distribution engineering function inference language language models large language large language models llms prompt rlhf safety sampling type via work