Switchable Decision: Dynamic Neural Generation Networks

arXiv:2405.04513v1 Announce Type: cross
Abstract: Auto-regressive generation models achieve competitive performance across many different NLP tasks such as summarization, question answering, and classifications. However, they are also known for being slow in inference, which makes them challenging to deploy in real-time applications. We propose a switchable decision to accelerate inference by dynamically assigning computation resources for each data instance. Automatically making decisions on where to skip and how to balance quality and computation cost with constrained optimization, our dynamic neural …

abstract applications arxiv auto computation cs.ai cs.cl cs.lg decision deploy dynamic however inference networks nlp performance question question answering real-time real-time applications summarization tasks them type