Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

When Andrej Karpathy left OpenAI last week a lot of people expressed hope that he would be increasing his output of educational YouTube videos.

Here's an in-depth 2 hour dive into how tokenizers work and how to build one from scratch, published this morning.

The section towards the end, "revisiting and explaining the quirks of LLM tokenization", helps explain a number of different LLM weaknesses - inability to reverse strings, confusion over arithmetic and even …

ai andrej karpathy andrejkarpathy build educational generativeai gpt hour llms openai people scratch videos work youtube