Tiktoken — AI Glossary

Reference details

Related terms

• Byte-Pair Encoding (BPE)
• Vocabulary Size
• Context Window

How it works

1. A named encoding is loaded, each corresponding to a specific vocabulary a model family was trained with.
2. Text is encoded greedily against that vocabulary's learned merges, producing exactly the token ids the model will receive.
3. Because the encoding is the model's own, the count is exact rather than an estimate — which is the reason to use it over a heuristic.
4. The core is compiled, so counting a large corpus is fast enough to run inline in a pipeline rather than as a batch job.
5. Choosing the wrong encoding for a model gives a confidently wrong count, since vocabularies differ and the function will happily encode with any of them.

Example: Why heuristics fail exactly where it matters

A four-characters-per-token rule of thumb estimates a 4,000-character prompt at 1,000 tokens. For ordinary English prose it lands within a few percent. For a prompt dense with identifiers, JSON punctuation or a non-Latin script, the real count can be 1.5 to 2.5 times that — so a request estimated at 1,000 tokens arrives as 2,200 and overflows a budget sized on the heuristic. The failure is not random: it concentrates on exactly the inputs that are unusual, which are the ones most likely to be near a limit.

Commonly confused with

Sometimes treated as the tokeniser, when it is one implementation serving one family's vocabularies. Other families use different tokenisers with different vocabularies, so a count from here is not a count for them — and the resulting error is systematic rather than small. It is also not a cost calculator: it gives exact input token counts, while output length is unknown before generation and reasoning tokens are billed without being shown, so a cost estimate needs both halves and this supplies one.

When to use it

Reach for it when:

• Counting input tokens exactly for a model whose encoding this provides.
• Enforcing a context budget before sending a request rather than discovering it on rejection.
• Measuring prompts dense with code, punctuation or non-Latin script, where heuristics diverge.

Reach for something else when:

• Counting for a model family with a different tokeniser, which gives a systematically wrong number.
• Estimating total cost, since output length is not known in advance.
• Where an approximation is sufficient and the text is ordinary English prose.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tiktoken — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path