Reference 2 stops to get here

Stemming

Reducing words to their root form by removing affixes (prefixes, suffixes, infixes), simpler than lemmatization but less linguistically accurate.

Your route here

2 stops · basics first
  1. Token ✓ understood

    The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.

  2. Tokenization ✓ understood

    Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.

  3. Stemming · you are here ✓ understood

Where it sits

Before this

Tokenization
Stemming

Leads to

Nothing yet: a destination in its own right.

Explore nearby