Tokenizer — AI Glossary

Reference details

Related terms

• Token
• Large Language Model

How it works

1. A vocabulary is learned once from a corpus, merging frequently co-occurring byte or character sequences into single units.
2. That vocabulary is fixed for the model's life, so the same text always maps to the same token ids for that model and to different ones for another.
3. Encoding is greedy over the learned merges, which is why a common word is one token and a rare one splits into several pieces.
4. Text outside the training distribution — other scripts, unusual formatting, long identifiers — fragments into many tokens because no merges cover it.
5. Every limit and price that mentions tokens is denominated in this model's units, so counts are not comparable across providers.

Example: Why the same text costs different amounts

A passage is 1,000 English words. At roughly 1.3 tokens per word that is about 1,300 tokens. The same content in a language the vocabulary covers poorly can run 2.5 tokens per word, about 2,500 — nearly double, for identical information. Against a 128,000-token window that is the difference between fitting 98 such passages and 51. Nothing about the model's capability changed; the vocabulary did, which is why multilingual cost and context planning cannot use an English token ratio.

Commonly confused with

Three related things get used interchangeably. The token is the unit, byte-pair encoding is one algorithm for deciding what the units are, and the tokenizer is the component that applies a specific learned vocabulary to turn text into IDs and back. Two tokenizers using the same algorithm still produce different token counts for the same text, so the vocabulary is the thing that has to match, not the algorithm.

When to use it

Reach for it when:

• Estimating cost or context usage, which must be done in the target model's own tokeniser.
• Explaining why token counts differ across providers for identical text.
• Debugging why a prompt that fitted one model overflows another.

Reach for something else when:

• As a universal unit — a token count is model-specific.
• Approximating by word or character count for non-English text or code, where the ratio diverges sharply.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tokenizer — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path