Andrej Karpathy's Llama 3 review

Andrej Karpathy's Llama 3 review

The most interesting coverage I've seen so far of Meta's Llama 3 models (8b and 70b so far, 400b promised later).

Andrej notes that Llama 3 trained on 15 trillion tokens - up from 2 trillion for Llama 2 - and they used that many even for the smaller 8b model, 75x more than the chinchilla scaling laws would suggest.

The tokenizer has also changed - they now use 128,000 tokens, up from 32,000. This …

70b ai andrej karpathy andrejkarpathy coverage generativeai llama llama 2 llama 3 llms meta notes review tokens