mistralai/mistral-common

mistralai/mistral-common

New from Mistral: mistral-common, an open source Python library providing "a set of tools to help you work with Mistral models".

So far that means a tokenizer! This is similar to OpenAI's tiktoken library in that it lets you run tokenization in your own code, which crucially means you can count the number of tokens that you are about to use - useful for cost estimates but also for cramming the maximum allowed tokens in the context window for …

ai anthropic code count generativeai library llms mistral openai open source promptengineering python rag set tokenization tokens tools work