With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation

arXiv:2401.11504v2 Announce Type: replace
Abstract: Long text generation, such as novel writing and discourse-level translation with extremely long contexts, presents significant challenges to current language models. Existing methods mainly focus on extending the model's context window through strategies like length extrapolation. However, these approaches demand substantial hardware resources during the training and/or inference phases. Our proposed method, Temp-Lora, introduces an alternative concept. Instead of relying on the KV cache to store all context information, we embeds this information directly into …

abstract arxiv challenges context context window cs.ai cs.cl current demand discourse focus hardware however inference language language models novel resources strategies text text generation through training translation type writing