Reference 7 stops to get here

Gradient Checkpointing

Trading computation for memory by recomputing activations during backprop instead of storing them.

Your route here

7 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A computational model inspired by biological neural networks, consisting of interconnected nodes (neurons) organized in layers that process information through weighted connections.

  3. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  4. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  5. Loss Function ✓ understood

    A function that scores how wrong a model's prediction is as a single number, which training then works to make as small as possible.

  6. Gradient Descent ✓ understood

    An optimization method that repeatedly moves a model's parameters a small step in the direction that most reduces the loss.

  7. Backpropagation ✓ understood

    The algorithm for computing gradients of the loss with respect to network weights, enabling training through gradient descent.

  8. Gradient Checkpointing · you are here ✓ understood

Trading computation for memory by recomputing activations during backprop instead of storing them.

This concept is essential for understanding training & optimization and forms a key part of modern AI systems.

  • Memory Efficiency
  • Training
  • Backpropagation

Where it sits

Before this

Backpropagation
Gradient Checkpointing

Leads to

Nothing yet: a destination in its own right.

Explore nearby

Training Gradient Accumulation Summing gradients over multiple batches before updating, simulating larger effective batch sizes. Training Mixed Precision Training Using lower precision (FP16) for some computations while keeping FP32 for stability, speeding up training. Training ZeRO Zero Redundancy Optimizer - techniques for memory-efficient distributed training by partitioning optimizer states. Language & LLMs Flash Attention An efficient attention algorithm reducing memory usage and increasing speed through clever recomputation. Neural Networks Forward Pass The process of passing input through the network to generate predictions during training or inference.