Token — AI Glossary

Reference details

Related terms

• Tokenizer
• Context Window
• Inference Cost

How it works

1. A tokenizer is trained, not written: byte-pair encoding starts from individual characters and repeatedly merges the most frequent adjacent pair, recording each merge until the vocabulary reaches its target size.
2. Frequent sequences therefore become single tokens while rare ones stay fragmented, which is why common English words cost one token and unusual ones cost several.
3. That frequency was measured on a particular corpus, so a script underrepresented in it never accumulated merges and its text fragments to near one token per character — making identical meaning cost several times more to send.
4. A leading space is part of the token rather than separate, so the same word costs differently at the start of a line than mid-sentence, and a stray double space changes the tokenisation of what follows.
5. A block of ids is reserved for special tokens — turn delimiters, end-of-sequence markers — and these have no spelling reachable from ordinary text, which is what prevents a user from typing a string that closes the system turn.
6. Digits are grouped by the same frequency-driven merges rather than individually, so the internal structure of a long number is split arbitrarily — which is the mechanism behind unreliable arithmetic on large values.

Example: The model cannot see the letters

English runs roughly 0.75 words per token, so a 500-word document is about 500 / 0.75 ≈ 667 tokens, and a 200,000-token context window holds on the order of 150,000 words. The more consequential fact is what a token *is*: the model receives integer IDs for word fragments, not characters. Asking how many times a letter appears in a word is therefore asking it to report on something it never received — it can often infer the answer, but it is inference, not inspection. The same mechanism explains why arithmetic on long numbers is unreliable: the digits were grouped into fragments before the model saw them.

Commonly confused with

A token is not a word, a syllable or a character, and the mapping is different for every tokenizer. Counts do not transfer between providers even for identical text, and languages not written in Latin script frequently cost several tokens per character — which makes the same meaning materially more expensive to send. Always count with the tokenizer of the model you are actually calling.

When to use it

Reach for it when:

• Every cost, latency and context-fit estimate — tokens are the unit all three are billed in
• Deciding chunk sizes, truncation points and how much history to carry
• Explaining character-level and arithmetic failures, which are usually tokenization artifacts

Reach for something else when:

• Estimating non-English cost from an English words-per-token ratio
• Assuming a provider's count matches another's, or that a rough estimate is close enough at the context limit
• Character-level tasks, which are better served by doing the string operation in code

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Token — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path