What Is a Token in AI? Tokens, Cost & Limits Explained

Article

Plain-English definition

A token is the unit of text a language model reads and writes. Before a model sees your prompt, a component called a tokenizer splits the text into tokens drawn from the model’s fixed vocabulary. Common words are usually one token ("the", "hello"); rarer words split into pieces ("tokenization" -> "token" + "ization"); punctuation and spaces count too.

A useful rule of thumb for English, drawn from the provider glossaries listed in Sources: one token is roughly 4 characters, or about three-quarters of a word. So 1,000 tokens is on the order of 750 words. Treat this as an estimate, not a law; the real ratio depends on the text and the model.

Why it matters

Tokens are the currency of AI systems, in four concrete ways:

- Cost. API pricing is quoted per million tokens, with input and output priced separately (output usually costs more). - Limits. The context window is a token count that decides how much fits in one request. - Speed. Models generate output token by token, so long answers take proportionally longer; throughput is quoted in tokens per second. - Output caps. Maximum response length is a token limit; hit it and the answer truncates mid-thought.

How tokenization works

Modern tokenizers are built by analyzing huge amounts of text and learning which character sequences occur most often. Most descend from byte-pair encoding, a method that builds a vocabulary by repeatedly merging the most frequent pairs of characters into larger units. Frequent sequences earn their own token; everything else is composed from smaller pieces. The result is a fixed vocabulary, ranging from tens of thousands to a few hundred thousand entries depending on the model family, that can represent any input in any language by falling back to smaller fragments when needed.

This is why token counts vary by model: each family learns its own vocabulary from its own training mix. It’s also why non-English text often costs more: languages underrepresented in training data fragment into more tokens per sentence.

Simple examples

The splits below are illustrative examples of how tokenizers typically behave, not universal tokenizer results; every model family splits text its own way.

English text. Take the sentence: "Unbelievably, the API responded instantly." A tokenizer might split it as: Un | believ | ably | , | the | API | responded | instantly | . That is nine tokens for six words: common words stay whole, rarer words split into pieces.

Non-English text. The same idea in another language often costs more tokens. A short Swahili sentence like "Habari za asubuhi" (good morning) may split into more, smaller fragments, for example Hab | ari | za | as | ub | uhi, because the language is less represented in the tokenizer’s training text. This is why identical meaning can bill differently across languages.

Code. Source code tokenizes differently again: `for (let i = 0; i < 10; i++)` is dense with punctuation and symbols, and each brace, semicolon, and operator can be its own token. Code files often use more tokens per visible character than prose, which is why coding assistants consume context quickly.

Now the billing math: sending a 3,000-word English document is roughly 4,000 input tokens by the rule of thumb above. At an illustrative price of $3 per million input tokens, reading that document costs about one cent, before the model writes a single output token.

Common misunderstandings

- "Tokens are words." Often they’re fragments. Word counts systematically underestimate token counts. - "Token counts are universal." They’re model-specific. Comparing model prices fairly means comparing on the same text through each model’s own tokenizer. - "Only my question costs tokens." System instructions, conversation history, retrieved documents, and the model’s own output all count, often far more than the question itself. - "Character limits and token limits are the same." Code and non-Latin scripts can tokenize very differently from prose; the same character count can be wildly different token counts.

Where tokens appear in real systems

Pricing pages and usage dashboards; context-window specs on model reference pages; rate limits (often tokens per minute); streaming UIs that render token by token; and cost engineering in RAG systems, where chunk sizes are chosen in tokens to budget the prompt.

Limitations and caveats

- Estimates drift. The three-quarters-word rule is calibrated on English prose; verify with the provider’s counting tools for your actual content. - Tokenizers evolve. New model generations sometimes ship new tokenizers, changing counts and costs for identical text. - Billed does not equal visible. Some features (reasoning traces, tool definitions) consume tokens you don’t see in the response.

Next steps

See what the token limit governs in what is a context window, then go hands-on with the tokenization tutorial. Compare real per-token pricing in the model comparisons.

Sources and further reading

- Anthropic, AI glossary (tokens) -- provider definition and context. - OpenAI, Models documentation -- context windows and token limits per model.