Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

arXiv:2310.16834v2 Announce Type: replace-cross
Abstract: Despite their groundbreaking performance for many generative modeling tasks, diffusion models have fallen short on discrete data domains such as natural language. Crucially, standard diffusion models rely on the well-established theory of score matching, but efforts to generalize this to discrete structures have not yielded the same empirical gains. In this work, we bridge this gap by proposing score entropy, a novel loss that naturally extends score matching to discrete spaces, integrates seamlessly to build …

abstract arxiv cs.cl cs.lg data diffusion diffusion modeling diffusion models distribution domains generative generative modeling groundbreaking language modeling natural natural language performance standard stat.ml tasks theory type