David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs

arXiv:2305.14771v2 Announce Type: replace
Abstract: Diffusion-based language models are emerging as a promising alternative to autoregressive LMs: they approach the competence of autoregressive LMs while offering nuanced controllability at inference time. While autoregressive LMs have benefited immensely from scaling and instruction-based learning, existing studies of diffusion LMs have been conducted on a smaller scale. Starting with a recently proposed diffusion model SSD-LM, in this work we first explore methods to scale it from 0.4B to 13B parameters, proposing techniques to …

abstract arxiv collaboration cs.cl david diffusion general inference language language models lms scaling small studies type