LongLoRA: New method extends LLAMA2 7B to 100k context length, 70B to 32k context length on on a single 8 × A100 machine

As AI models get bigger, training them requires more and more computing power. Researchers are looking for ways to train these large AI models without needing Google-scale resources.

A new paper proposes [LongLoRA](https://arxiv.org/pdf/2309.12307.pdf), **a fine-tuning approach that can extend LLaMA2 7B to 100k context length and 70B model to 32k context length on a single 8× A100 machine.**

Here are my highlights from the paper:

Big one of course: LongLoRA efficiently fine-tunes large AI models on longer texts

Key points: …

32k context a100 ai models artificial big bigger computing computing power context google highlights llama2 machine paper power researchers resources scale them training