Reference 2 stops to get here

Vocabulary Size

The number of distinct tokens a language model can process, typically 30K-100K+ tokens.

Your route here

2 stops · basics first
  1. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  2. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  3. Vocabulary Size · you are here ✓ understood

The number of distinct tokens a language model can process, typically 30K-100K+ tokens.

This concept is essential for understanding large language models and forms a key part of modern AI systems.

  • Tokenization
  • Token
  • Embedding

Where it sits

Before this

Tokenization
Vocabulary Size

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Language & LLMs Subword Tokenization Breaking words into smaller units, balancing vocabulary size with representation granularity. Language & LLMs BPE Byte Pair Encoding - a subword tokenization algorithm that iteratively merges frequent character pairs to create a vocabulary. Language & LLMs Embedding A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together. Language & LLMs Token The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.