Standard 3 stops to get here · leads to 1

Distributed Training

Training one model across many GPUs or machines at once, by splitting the data, the model, or both, and keeping every copy in sync.

Your route here

3 stops · basics first
  1. GPU ✓ understood

    Graphics Processing Unit - hardware accelerator with thousands of cores, essential for parallel computation in deep learning.

  2. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  3. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  4. Distributed Training · you are here ✓ understood

Picture it

Data parallelism

  • Every GPU holds the whole model
  • Each sees a different slice of the batch
  • Gradients are averaged every step

Model parallelism

  • The model itself is split up
  • By layer (pipeline) or within a layer
  • Activations pass between GPUs

Sharded states

  • Replicas split optimizer state
  • Parameters gathered when needed
  • Data-parallel speed, less memory
Large training runs mix all three: split the data to go faster, split the model to fit it, and shard the states so nothing is stored twice.

Distributed training spreads the work of training a single model across many accelerators, from a few GPUs in one server to thousands across a cluster. It exists for two reasons: a model can take too long to train on one device, or be too big to fit on one at all. Today’s large language models need both kinds of help.

Three ways to split the work

  • Data parallelism: every device holds a full copy of the model and processes a different slice of each batch. After the backward pass the devices average their gradients, so every copy takes the same step. It speeds training up but doesn’t help a model that won’t fit.
  • Model parallelism: the model itself is divided. Pipeline parallelism gives each device a run of layers; tensor parallelism splits the matrix multiplications inside each layer. Megatron-LM used the second to train an 8.3-billion-parameter language model on 512 GPUs.
  • Sharding: plain data parallelism stores the weights, gradients and optimizer state on every device. ZeRO and similar schemes partition those states across devices instead, removing the duplication while keeping data parallelism’s simplicity.

Large runs usually combine all three, for example tensor parallelism inside a server, pipeline stages across servers, and data parallelism across groups of servers.

The catch

Devices must constantly exchange gradients, weights or activations, and that communication competes with computation. Adding hardware helps only as long as the network between devices keeps up, which is why the interconnect is as important as the chips. Failures matter too: across thousands of devices, something breaks often enough that long runs rely on regular checkpoints to resume.

Where it sits

Before this

Training GPU
Distributed Training

Leads to

Data Parallelism

Explore nearby

Training Data Parallelism Replicating the model across devices, each processing different data batches. Training Tensor Parallelism Splitting individual layers/tensors across devices for very large models. Training Pipeline Parallelism Splitting model layers across devices and processing micro-batches in pipeline fashion. Training Sharded Data Parallelism Distributing model states across devices to train models larger than single-device memory. Training ZeRO Zero Redundancy Optimizer - techniques for memory-efficient distributed training by partitioning optimizer states. Training Gradient Accumulation Summing gradients over multiple batches before updating, simulating larger effective batch sizes.

Sources

  1. Mohammad Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" . arXiv, 2019
  2. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase and Yuxiong He, "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models" . SC 2020