GIVT: Generative Infinite-Vocabulary Transformers

arXiv:2312.02116v3 Announce Type: replace
Abstract: We introduce generative infinite-vocabulary transformers (GIVT) which generate vector sequences with real-valued entries, instead of discrete tokens from a finite vocabulary. To this end, we propose two surprisingly simple modifications to decoder-only transformers: 1) at the input, we replace the finite-vocabulary lookup table with a linear projection of the input vectors; and 2) at the output, we replace the logits prediction (usually mapped to a categorical distribution) with the parameters of a multivariate Gaussian mixture …

abstract arxiv cs.cv decoder generate generative linear projection simple table tokens transformers type vector vectors