Lab 01
Token Visualiser
Models do not read words. Text is cut into tokens first, and tokens are what the context limit and the bill are measured in.
Approximate. This splits text using rules that mirror how real tokenisers behave, but it is not any specific model's tokeniser. Real vocabularies are learned from data and differ between models — treat the count as an illustration, not a billing estimate.
Your text
Try
Tokens (13)
Explain·artificial·intelligence·in·simple·words.
- Word
- Word piece
- Punctuation
Counts
- Tokens
- 13
- Characters
- 48
- Words
- 6
- Chars / token
- 3.69
Share of a context window
0.32% of this window
These sizes are illustrative round numbers for comparing scale, not the limits of any particular product.
Reading the result
- What just happened
- Your text was cut into 13 pieces. Notice that the cuts do not line up with words — long words break apart, and the space before a word travels with it.
- Why it matters
- Tokens are the unit everything is measured in: how much you can send, how much a model can remember at once, and what you are charged.
- Where you meet this
- Every time you paste a long document into a chat and it says the conversation is too long, this is the arithmetic behind it. It is also why the same request can cost noticeably more in one language than another.