Landmark A starting point · leads to 7

Token

The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

Picture it

Word-level

  • "unhappiness" → 1 token
  • Huge vocabulary
  • Unknown words become [UNK]

Subword (BPE)

  • "unhappiness" → "un" + "happiness"
  • What most modern LLMs use
  • Handles rare words from known pieces

Character-level

  • "unhappiness" → 11 tokens
  • Tiny vocabulary
  • Long sequences, costly to process
Notice the trade-off: bigger tokens mean shorter sequences but larger vocabularies, and subwords sit in the middle, which is why LLMs use them.

A token is the unit of text a language model actually reads and writes. Before a model sees your prompt, a tokenizer chops it into pieces from a fixed list, its vocabulary, and swaps each piece for an ID number. The model never sees letters or words, only a sequence of these IDs, and it generates its answer one token at a time.

A token can be a whole word, part of a word, a single character, a space-plus-word, or a punctuation mark. Common words are usually one token each; rare words, names and code get split into several.

Why subwords won

The diagram compares the three ways to cut text. Whole words give short sequences but need an enormous vocabulary, and any word missing from it becomes an unknown. Single characters never run into unknown words but make sequences very long, and every extra token costs compute. Subwords sit in between.

The approach took hold in 2015, when Sennrich and colleagues adapted byte pair encoding to translation. BPE starts from characters and repeatedly merges the most frequent adjacent pair into a new symbol, so frequent words end up as single tokens while rare ones are built from known pieces. WordPiece, used by BERT, is a close relative. SentencePiece (2018) trains directly on raw text, treating spaces as just another symbol, which makes it work for languages that don’t separate words with spaces.

Once text is tokenized, each ID is looked up in a table of learned vectors, the model’s embeddings. Tokenizers also reserve special tokens for things like the start of a sequence or the end of a turn.

Why tokens matter in practice

Nearly everything about using a large language model is measured in tokens:

  • Limits. The context window is a token budget covering the prompt, any documents and the reply.
  • Cost. Commercial APIs price input and output per token, often with output tokens priced higher.
  • Speed. Generation happens one token per step, so longer answers take proportionally longer.

The Duel turns that budget into a game: watch a context window fill up and see what it costs.

The catch

The same text can be a different number of tokens under different tokenizers, so counts don’t transfer between model families. The same sentence translated into another language can take many times more tokens, which makes some languages slower and more expensive to use. And because models see tokens rather than letters, tasks like counting the letters in a word can trip them up.

Where it sits

Before this

Nothing: this is a starting point.

Token

Explore nearby

In the research

All papers →

2 papers that build on Token .

Sources

  1. Sennrich, Haddow and Birch, "Neural Machine Translation of Rare Words with Subword Units" . 2015; ACL 2016 (byte pair encoding for subwords)
  2. Kudo and Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" . 2018
  3. Petrov et al., "Language Model Tokenizers Introduce Unfairness Between Languages" . 2023