What are tokens?
Models do not see words. They see tokens, the fragments that pricing, context limits, and speed are all measured in.
The short version
- → A token is a chunk of text, often a word fragment or a common word.
- → A rough rule of thumb: one token is about four characters, or three quarters of a word.
- → Tokens are the unit for pricing, context limits, and how fast responses stream.
Language models do not process whole words or individual letters. They break text into tokens, pieces that are usually part of a word, a short whole word, or a punctuation mark. "Unbelievable" might become "un", "bel", "iev", "able".
This sounds like a technical detail, but it is the unit everything else is measured in. Cost, context limits, and speed are all counted in tokens, not words.
Why split into tokens at all
A fixed vocabulary of full words would be huge and would still miss new or rare words. Splitting into common fragments keeps the vocabulary manageable while letting the model build any word, including ones it never saw, from pieces. It is a practical compromise between letters and whole words.
The rough conversions
For English, one token averages about four characters, and 100 tokens is roughly 75 words. So a page of text is around 500 tokens, and a long document can be thousands. Code, other languages, and unusual formatting tokenize differently, sometimes far less efficiently.
Why you are billed in tokens
Every token, in your prompt and in the reply, has to be processed through the whole network, so tokens map directly to compute. That is why providers price per token and count both directions. A long back-and-forth costs more because the whole conversation is re-processed as context each turn.
Tokens set the limits
The context window, the maximum the model can consider at once, is measured in tokens. Generation speed is often quoted in tokens per second. Once you think in tokens, the model's pricing, limits, and pacing all line up.
An analogy
Think of tokens as the syllables a model reads and speaks in. It does not take in whole words at a glance or spell letter by letter. It works in these in-between chunks, and it counts them constantly.
Go deeper
Embeddings
By the end you will know how a token becomes a vector, what "close" means between two vectors, how the numbers are learned, and what similarity can and cannot tell you. 17 min read →
Where Berges AI fits
You never have to count tokens to use Berges AI, but they explain what you feel: why a message is capped at 4,000 characters, and why a concise prompt gets a faster answer. Trimming filler is good for you and the model.
Related concepts
What is an LLM?
The kind of AI behind chat assistants, explained without the jargon: what it is, how it produces text, and where it falls short. Read →
What is a context window?
The model's working memory: how much text it can hold in mind at once, and why it seems to forget once you go past it. Read →
How transformers work
The neural network design that powers almost every modern language model, and the one idea, attention, that made it work. Read →
Things people ask
How many words is a token?
On average, one token is about three quarters of a word in English, or roughly four characters. So 1,000 tokens is around 750 words. It varies with the exact text.
Do spaces and punctuation count as tokens?
Yes. Spaces are usually attached to the following word, and punctuation marks are often their own tokens. Everything in the text contributes to the count.
Why does the same text cost more in another language?
Tokenizers are usually optimized for English. Other languages and scripts can break into more tokens per word, so the same meaning costs more tokens, and therefore more money and context.