Standard 4 stops to get here · leads to 4
Autoregressive Model
A model that generates output one token at a time, using previously generated tokens as input for the next prediction.
Your route here
4 stops · basics first
- Natural Language Processing ✓ understood
The field of AI that lets computers read, interpret, translate and generate human language, from spam filters and search to chatbots.
- Token ✓ understood
The unit of text a language model reads and writes: a word, part of a word, or a character, drawn from the model's fixed vocabulary.
- Tokenization ✓ understood
Splitting text into tokens, usually subword pieces, and mapping each to an integer ID so a language model can process it.
- Language Modeling ✓ understood
Assigning probabilities to sequences of text, in practice by predicting each next token from the ones before it; the objective behind large language models.
- Autoregressive Model · you are here ✓ understood
Where it sits
Before this
Language Modeling Autoregressive Model
Explore nearby
Language & LLMs Next-Token Prediction The task of predicting the next token in a sequence from all the tokens before it: the training objective behind GPT-style language models and how they generate text. Language & LLMs GPT Generative Pre-trained Transformer - an autoregressive language model architecture that predicts the next token given previous context. Language & LLMs Causal Language Modeling Training a model to predict the next token given previous tokens, the foundation of autoregressive models like GPT. Language & LLMs Decoder-Only Model A transformer architecture with only decoder layers, using causal masking for autoregressive generation (GPT family). Language & LLMs Text Generation Automatically creating coherent text using language models, from simple completion to creative writing. Language & LLMs Causal Mask An attention mask ensuring tokens can only attend to previous positions, crucial for autoregressive generation.
In the research
All papers →6 papers that build on Autoregressive Model ; showing 5, canon first.