LLama-2 7B: 400K context length - Beyond Limits?

A LLama-2 7B LLM with 400K context length has been build with a new method. Based on activation compression and activation beacons for context intervals. With a sliding window methodology a context length of 400K for a LLama-2 7B LLM has been tested. Here are the results.

A new competitive method to extend context lengths of LLM not just by fine-tuning? Is it useful to extend your LLM from 4K to maybe just 32K context length? What compute infrastructure do …

beacons beyond build compression context llama llm methodology