The Tokenizer Playground

The Tokenizer Playground

I built a tool like this a while ago, but this one is much better: it provides an interface for experimenting with tokenizers from a wide range of model architectures, including Llama, Claude, Mistral and Grok-1 - all running in the browser using Transformers.js.

Via @xenovacom

ai architectures browser claude generativeai grok llama llms mistral playground running the browser tool transformers via