Tokens: The Atoms of Language Models
How AI reads, how it writes, and why a 200K-token context window in English is not the same as a 200K-token context window in Japanese.
*These are learning tools, not prompts. They teach you to write your own, think of them as training wheels that guide you while you learn, then fall away once you can ride on your own.
A Token Is the Unit a Model Reads and Writes
A token is not a word, and it is not a letter. It is a chunk of text somewhere in between, the smallest piece of language a model actually processes. The sentence “The quick brown fox” is four tokens. The single word “antidisestablishmentarianism” is six. Common words usually fit into one token. Rare words, technical jargon, and most non-English text get broken into multiple smaller pieces.
The model never sees your letters. It sees integer IDs that refer to tokens in its vocabulary, somewhere in the range of fifty thousand to a few hundred thousand entries depending on the model. Every word you type is converted to those IDs before the model runs, and the model’s answer comes back as IDs that get converted back into text for you to read.
This matters because everything downstream of that conversion, attention, prediction, pricing, rate limits, context windows, operates on token IDs. When you understand tokens, you understand why prompts cost what they cost, why “200K context” means different things in different languages, and why models cannot count letters in “strawberry.”
A token is a chunk of text the model treats as a single unit. Not a word, not a letter, somewhere in between, and assigned an integer ID.
Vocabulary Economics and Compute Reality
A model needs a vocabulary, a fixed list of every unit it can read or generate. Two extremes do not work. A pure-word vocabulary would need millions of entries and would still miss every typo, brand name, and word coined after training. A pure-character vocabulary would be tiny but would force the model to relearn spelling for every sequence, wasting most of its capacity on letter-shaping instead of meaning.
Subword tokenization is the engineering compromise. A vocabulary of fifty thousand to a few hundred thousand subword pieces covers essentially any input, common words stay whole as single tokens, rare words split into recognizable parts, and the model never encounters an unknown unit.
There is also a compute reason. Self-attention scales with the square of the token count, every token attends to every other token, so the attention matrix is n by n. For short prompts this is not the dominant cost, the feedforward block in each layer dominates total compute below roughly eight thousand tokens. Past that the quadratic regime takes over and really bites at thirty-two thousand and beyond. Tokenization is one of the engineering reasons modern long-context models are economically possible at all.
Self-attention is O(n²), so at long contexts the cost of doubling your prompt approaches 4× rather than 2×. The quadratic regime dominates once you cross roughly eight thousand tokens. If you are running your own inference at scale on long contexts, every token saved is real money. If you are paying an API per token, your savings are linear regardless of context length, because the provider absorbs the n² cost in their infrastructure economics.
Byte-Pair Encoding in Plain Steps
Almost every modern language model uses a flavor of subword tokenization. The most common algorithm is Byte-Pair Encoding, originally a data-compression technique, repurposed for language. Here is how it builds its vocabulary, step by step.
- Start with characters. The initial vocabulary is every individual character in the training data, letters, digits, punctuation, whitespace.
- Count adjacent pairs. Across the entire training corpus, count how often each pair of adjacent tokens appears.
- Merge the most frequent pair. If
tandhappear next to each other more often than any other pair, create a new tokenthand add it to the vocabulary. - Repeat. Recount pairs (the new
thtoken now participates), merge the next most frequent pair, again and again, until the vocabulary reaches the target size. - Save the merge table. At inference time, your input text is processed left-to-right against this learned merge table to produce the same token sequence the model was trained on.
BPE is the lineage behind GPT-2, GPT-3, GPT-4, and most modern open-weight models. Two close cousins use the same idea with different rules. WordPiece (BERT, DistilBERT) merges by likelihood gain instead of raw frequency. SentencePiece (T5, LLaMA, modern Llama family) treats the input as a raw byte stream including whitespace, which makes it easier to handle languages without space-separated words.
You can paste any text into OpenAI’s tokenizer to watch this split happen on real input, character by character.
Most production tokenizers first split on whitespace and punctuation, then apply BPE merges inside each piece. This is why “tokenizer” might be one token but “tokenizer.” (with the period) becomes two.
Tokenization Is Not Language-Neutral
The same idea costs different numbers of tokens depending on the script.
| Text | Characters | Tokens | Density (chars/token) |
|---|---|---|---|
Hello, world. |
13 | 4 | 3.25 |
Hola, mundo. |
12 | 4 | 3.00 |
こんにちは、世界。 |
9 | 4 | 2.25 |
def hello(): print("hi") |
24 | 7 | 3.43 |
Counts measured against OpenAI’s current o200k_base encoding. Modern tokenizers have narrowed the historical cost gap between scripts. English still leads on density, Japanese still trails, but the gap for short phrases is closer to thirty percent than the three-to-four-times multiplier older encodings produced. For longer text, rare vocabulary, or technical content, the gap widens again. Tokenizer choice and content type both matter.
When Tokens Matter to You
Four places tokens stop being theory and start affecting real work.
Cost
APIs price input tokens and output tokens separately, and output is typically two to five times more expensive than input. A long system prompt is billed on every single call. Conversation history is billed every turn, growing linearly until you trim it. Cost discipline is mostly token discipline.
Context Windows
A “200K-token context” is a token budget, not a character budget. In English that is roughly 150,000 words. Other scripts hold less raw text in the same budget, modern tokenizers have narrowed the gap from earlier generations, but the point still stands. Multilingual applications must size context for the worst-case language and content type.
Rate Limits
Production deployments hit tokens-per-minute ceilings long before they hit requests-per-minute. TPM is the real production governor. Bursts of long requests can throttle your application before a traffic spike does.
The Strawberry Problem
Models often miscount letters in words because they do not see letters. They see the token straw and the token berry. The famous “how many R’s in strawberry” failure is not a model being dumb, it is a model being asked to inspect a representation it does not have direct access to.
Three Mistakes to Avoid
Common assumptions that quietly cost you accuracy, money, or capacity.
The rule of thumb for English is around three-quarters of a word per token, not one. “Hello” is one token. “Hello, world!” is four. “antidisestablishmentarianism” is six. Punctuation, capitalization, and leading spaces all change the split. Trust the tokenizer, not your intuition.
Your system prompt is billed on every call. A 500-token system prompt across a million API calls is 500 million input tokens. If you are paying per token, you are paying for that system prompt every time. Treat it like a fixed monthly cost, not a free intro.
Character counts (len(string), string.length) are a rough estimate at best, and they fail completely for non-Latin scripts and code. The only number that matters for cost, context windows, and rate limits is the token count. If you need to budget accurately, call a tokenizer.
Keep Going
Build on what you just learned.
Prompt Basics
If tokens are the atoms, prompts are the molecules. Learn the four elements every effective prompt combines.
Glossary
Look up Token, Tokenization, Tokenizer, and 5,300+ other AI terms in plain language.
AI History
How tokenization fits into the seventy-five-year arc from Turing’s question to the agentic frontier.