Dodo: Dynamic Contextual Compression for Decoder-only LMs

arXiv:2310.02409v2 Announce Type: replace
Abstract: Transformer-based language models (LMs) are inefficient in long contexts. We propose Dodo, a solution for context compression. Instead of one vector per token in a standard transformer model, Dodo represents text with a dynamic number of hidden states at each layer, reducing the cost of self-attention to a fraction of typical time and space. Moreover, off-the-shelf models such as LLaMA can be adapted to Dodo by efficient parameter tuning methods such as LoRA. In use, …

abstract arxiv attention compression context cost cs.ai cs.cl cs.lg decoder dynamic hidden language language models layer lms per replace self-attention solution standard text token transformer transformer model type vector