Shipping AI May 2022

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Tri Dao et al. · NeurIPS 2022

arXiv:2205.14135

In short

Attention is slow mostly because of memory traffic, not arithmetic. FlashAttention computes exact attention in tiles that stay in the GPU’s fast on-chip memory, cutting reads and writes to main memory and speeding up training and inference.

Why it matters

It is a big part of why long context windows became affordable, and it ships inside nearly every LLM stack.

Read first

The 3 Field Guide ideas this paper leans on.

Starting from scratch? The full route 11 ideas · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  4. Feature ✓ understood

    A single measurable property of an example, such as a house's floor area or how many links an email contains, used as an input to a model.

  5. Deep Learning ✓ understood

    A subset of machine learning that uses neural networks with multiple layers (deep neural networks) to learn hierarchical representations of data.

  6. Representation Learning ✓ understood

    Letting a model learn its own internal description of the data, such as embeddings or layered visual features, instead of relying on features designed by hand.

  7. Embedding ✓ understood

    A list of numbers (a vector) that represents a word, sentence, image or other item, learned so that similar items end up close together.

  8. Attention Mechanism ✓ understood

    A technique that lets a neural network weigh every part of its input when producing each output, focusing on the parts most relevant at that step.

  9. Self-Attention · read first ✓ understood

    An attention step in which every token in a sequence looks at every token in that same sequence and builds a new representation from a weighted mix of them.

  10. GPU · read first ✓ understood

    Graphics Processing Unit - hardware accelerator with thousands of cores, essential for parallel computation in deep learning.

  11. Flash Attention · read first ✓ understood

    An efficient attention algorithm reducing memory usage and increasing speed through clever recomputation.

Nearby papers

Summary in our own words; read the paper for the details. ← All papers