TEncDM: Understanding the Properties of Diffusion Model in the Space of Language Model Encodings

arXiv:2402.19097v1 Announce Type: new
Abstract: Drawing inspiration from the success of diffusion models in various domains, numerous research papers proposed methods for adapting them to text data. Despite these efforts, none of them has managed to achieve the quality of the large language models. In this paper, we conduct a comprehensive analysis of key components of the text diffusion models and introduce a novel approach named Text Encoding Diffusion Model (TEncDM). Instead of the commonly used token embedding space, we …

abstract arxiv cs.cl data diffusion diffusion model diffusion models domains inspiration language language model language models large language large language models managed papers quality research research papers space success text them type understanding