How fast is JS tiktoken?

IMO, it is very fast, considering the performance against the OpenAI's Rust implementation (wrapped into a Python package). Both pure JS and Web Assembly (WASM) ports have decent speed, even interpreted JS version is within 10% margin on small and medium texts.

Below you can find execution times for text to tokens encoding (smaller is better).

• Small text (68 tokens)
  

Python/Rust (tiktoken 0.5.2)     ████ (0.04ms)
Pure JS (js-tiktoken 1.0.8)      █████ (0.05ms)
JS/WASM (tiktoken 1.0.11)        ██████████ (0.11ms)
@dqbd/WASM 1.0.7                 ██████████████████ (0.18ms) …

ai assembly encoding implementation interpreted javascript llm machinelearning medium openai package performance python rust small speed text tokens wasm web web assembly