Probabilistically-sound beam search with masked language models

arXiv:2402.15020v1 Announce Type: new
Abstract: Beam search with masked language models (MLMs) is challenging in part because joint probability distributions over sequences are not readily available, unlike for autoregressive models. Nevertheless, estimating such distributions has applications in many domains, including protein engineering and ancient text restoration. We present probabilistically-sound methods for beam search with MLMs. First, we clarify the conditions under which it is theoretically sound to perform text infilling with MLMs using standard beam search. When these conditions fail, …

abstract applications arxiv autoregressive models cs.cl cs.lg domains engineering language language models part probability protein protein engineering search sound text type