HASZB_AIHASZB_AI

Search across courses, lessons, glossary terms, prompts and tools.

Menu

Stage 2 · Understanding LLMs

Tokens — what a model actually reads

Models do not see words or letters. Understanding what they do see explains several behaviours that otherwise look like bugs.

6 min read

In this lesson

  • Explain what a token is
  • Predict roughly how text splits into tokens
  • Connect tokenisation to cost, limits and spelling mistakes

A model does not read your text as words, and definitely not as letters. Before anything happens, your text is cut into tokens.

What a token is

A token is a frequently-occurring chunk of characters. Tokenisers are built by analysing large amounts of text and finding which sequences appear often enough to deserve their own symbol.

The practical result:

  • Common words are usually one token: the, model, learning
  • Rarer or longer words split: tokenisation might become token + isation
  • Spaces usually attach to the following word: model is a different token from model
  • Punctuation is its own token
  • Unusual names, code and other languages fragment much more

A rough rule for ordinary English prose: about 4 characters per token, so 100 tokens is roughly 75 words. Treat that as an estimate, not a formula.

Different models, different splits

There is no single universal tokeniser. Different model families use different vocabularies, so the exact same sentence can produce different token counts depending on which model reads it.

This matters when you see a specific number quoted. A token count is only meaningful alongside the tokeniser that produced it.

Three things this explains

Cost and limits are measured in tokens. Providers bill per token and cap context in tokens. So a page of dense code costs more than a page of plain English, because it fragments into more tokens.

Non-English text is often more expensive. Tokenisers trained mostly on English represent English efficiently. The same meaning in a language with different script can take substantially more tokens.

Letter-level questions are unreliable. Asking how many times a letter appears in a word is asking about a level of detail the model does not directly manipulate. It has seen enough text about spelling to often get it right, but it is working against its own representation. This is why such puzzles fail in ways that look absurd next to the model's other abilities.

The useful intuition

Whenever a model behaves strangely with something character-level — spelling, reversing text, counting letters, rhyming in an unusual language — ask what the tokens probably look like. The answer is usually there.