A token is the unit of text a language model actually reads and writes. Before a model sees your prompt, a tokenizer chops it into pieces from a fixed list, its vocabulary, and swaps each piece for an ID number. The model never sees letters or words, only a sequence of these IDs, and it generates its answer one token at a time.
A token can be a whole word, part of a word, a single character, a space-plus-word, or a punctuation mark. Common words are usually one token each; rare words, names and code get split into several.
Why subwords won
The diagram compares the three ways to cut text. Whole words give short sequences but need an enormous vocabulary, and any word missing from it becomes an unknown. Single characters never run into unknown words but make sequences very long, and every extra token costs compute. Subwords sit in between.
The approach took hold in 2015, when Sennrich and colleagues adapted byte pair encoding to translation. BPE starts from characters and repeatedly merges the most frequent adjacent pair into a new symbol, so frequent words end up as single tokens while rare ones are built from known pieces. WordPiece, used by BERT, is a close relative. SentencePiece (2018) trains directly on raw text, treating spaces as just another symbol, which makes it work for languages that don’t separate words with spaces.
Once text is tokenized, each ID is looked up in a table of learned vectors, the model’s embeddings. Tokenizers also reserve special tokens for things like the start of a sequence or the end of a turn.
Why tokens matter in practice
Nearly everything about using a large language model is measured in tokens:
- Limits. The context window is a token budget covering the prompt, any documents and the reply.
- Cost. Commercial APIs price input and output per token, often with output tokens priced higher.
- Speed. Generation happens one token per step, so longer answers take proportionally longer.
The Duel turns that budget into a game: watch a context window fill up and see what it costs.
The catch
The same text can be a different number of tokens under different tokenizers, so counts don’t transfer between model families. The same sentence translated into another language can take many times more tokens, which makes some languages slower and more expensive to use. And because models see tokens rather than letters, tasks like counting the letters in a word can trip them up.