all AI news
[R] Adapting Language Models to Compress Contexts
May 31, 2023, 11:27 a.m. | /u/Balance-
Machine Learning www.reddit.com
[Alexis Chevalier](https://arxiv.org/search/cs?searchtype=author&query=Chevalier%2C+A), [Alexander Wettig](https://arxiv.org/search/cs?searchtype=author&query=Wettig%2C+A), [Anirudh Ajith](https://arxiv.org/search/cs?searchtype=author&query=Ajith%2C+A), [Danqi Chen](https://arxiv.org/search/cs?searchtype=author&query=Chen%2C+D)
>Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents …
computational context context window cost documents language language models machinelearning processing prompts summary text tools transformer vectors
More from www.reddit.com / Machine Learning
Jobs in AI, ML, Big Data
Data Scientist (m/f/x/d)
@ Symanto Research GmbH & Co. KG | Spain, Germany
Future Opportunity: Managed Services, Data Analyst
@ project44 | Poland - Kraków
Staff Software Engineer, Data Migration
@ Okta | Spain
Data Engineer
@ Red Bull | Thalgau, Austria
Head of Artificial Intelligence & Automation Transformation
@ Guardian | New York
Data Scientist-1
@ Visa | Bengaluru, India