Amortizing intractable inference in large language models

arXiv:2310.04363v2 Announce Type: replace
Abstract: Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest -- including sequence continuation, infilling, and other forms of constrained generation -- involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement …

abstract arxiv cs.cl cs.lg data forms however inference knowledge language language models large language large language models llms next posterior sampling tasks through token tractable training training data type