Tiktoken

Tiktoken is OpenAI’s fast, open-source byte pair encoding (BPE) tokenizer library for working with its language models. It converts text into the token sequences that models such as GPT-4o actually read, and back again, several times faster than comparable open-source tokenizers. Written in Rust with Python bindings, it has become a standard utility in the LLM ecosystem and has earned about 19,500 stars on GitHub.

Overview

Language models do not see raw text; they see tokens, and accurate tokenization matters for counting costs, fitting context windows and preparing training data. Tiktoken ships the exact encodings used by OpenAI models, including o200k_base for GPT-4o-class models and cl100k_base for GPT-4 and GPT-3.5 Turbo, and can look up the right encoding for a given model name automatically. Its BPE implementation is reversible and lossless, works on arbitrary text, and compresses text to roughly one token per four bytes of English. The library is also extensible: you can build custom Encoding objects or register new encodings through its plugin mechanism.

Key Features

• Official BPE encodings for OpenAI models, including o200k_base and cl100k_base
• Three to six times faster than comparable open-source tokenizers, thanks to a Rust core
• Simple API: get an encoding by name or automatically from a model name
• Lossless, reversible tokenization that handles arbitrary and out-of-vocabulary text
• Extensible design with custom Encoding objects and a plugin registration system
• Educational submodule for learning and visualizing how BPE tokenization works

💳 Pricing

Tiktoken is completely free and open source under the MIT license. It is a local library, so there are no usage fees beyond whatever you already pay for model APIs.

Who Is It For

Tiktoken is for developers building on OpenAI models who need fast, exact token counting and encoding for cost estimation, context management, prompt engineering and data pipelines.