Encode Once and Decode in Parallel: Efficient Transformer Decoding

arXiv:2403.13112v1 Announce Type: new
Abstract: Transformer-based NLP models are powerful but have high computational costs that limit deployment scenarios. Finetuned encoder-decoder models are popular in specialized domains and can outperform larger more generalized decoder-only models, such as GPT-4. We introduce a new configuration for encoder-decoder models that improves efficiency on structured output and question-answering tasks where multiple outputs are required of a single input. Our method, prompt-in-decoder (PiD), encodes the input once and decodes output in parallel, boosting both training …

abstract arxiv computational costs cs.cl decode decoder decoding deployment domains efficiency encode encoder encoder-decoder generalized gpt gpt-4 nlp nlp models popular transformer type