How to train a Million Context LLM — with Mark Huang of Gradient.ai

Scaling Llama3 beyond 1M context window with ~perfect utilization, the difference between ALiBi and RoPE, how to use GPT-4 to create synthetic data for your context extension finetunes, and more!

beyond context context window create data difference extension gpt gpt-4 gradient huang llama3 llm mark rope scaling synthetic synthetic data train