Reference 3 stops to get here

EOS Token

End Of Sequence token - signals when the model has finished generating a complete output.

Your route here

3 stops · basics first
  1. Token ✓ understood

    The basic unit of text that a language model processes, typically representing a word, subword, or character. Tokens are the fundamental building blocks for LLM input and output.

  2. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  3. Special Token ✓ understood

    Reserved tokens with special meanings like [CLS], [SEP], [MASK], [PAD] used in various model architectures.

  4. EOS Token · you are here ✓ understood

End Of Sequence token - signals when the model has finished generating a complete output.

This concept is essential for understanding large language models and forms a key part of modern AI systems.

  • Stop Token
  • Generation
  • Special Token

Where it sits

Before this

Special Token
EOS Token

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Language & LLMs BOS Token Beginning Of Sequence token - marks the start of a sequence in language models. Language & LLMs Stop Token A special token signaling the end of generation, causing the model to stop producing more tokens. Language & LLMs Padding Token A special token used to make sequences the same length in a batch, typically ignored during computation. Language & LLMs Text Generation Automatically creating coherent text using language models, from simple completion to creative writing.