Standard 6 stops to get here · leads to 2

Model Parallelism

Splitting a single model across several devices, by layers or within layers, so that a model too large for one accelerator can still be trained and run.

Your route here

6 stops · basics first
  1. GPU ✓ understood

    Graphics Processing Unit - hardware accelerator with thousands of cores, essential for parallel computation in deep learning.

  2. Dataset ✓ understood

    A collection of data examples used for training, validating, or testing machine learning models.

  3. Training ✓ understood

    The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.

  4. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  5. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  6. Layer ✓ understood

    A collection of neurons/operations that process data together, neural networks are composed of stacked layers.

  7. Model Parallelism · you are here ✓ understood

Picture it

Pipeline (by layer)

  • GPU 1 runs layers 1–12, GPU 2 runs 13–24
  • Activations hand off between stages
  • Micro-batches keep every stage busy

Tensor (within a layer)

  • Each weight matrix is split across GPUs
  • Every GPU computes part of each layer
  • Needs fast links: syncs inside each layer
Both split one model across devices; they differ in where the cut goes, and so in how often the devices must talk.

Model parallelism divides one model across several devices, so each GPU holds and computes only part of it. It is the answer to a model whose weights, gradients and activations don’t fit in one accelerator’s memory. Contrast data parallelism, where every device holds the whole model and only the data is split.

Where to cut

  • By layer: pipeline parallelism. Consecutive layers go to different devices, and activations flow from one stage to the next like an assembly line. Run naively, only one stage works at a time. GPipe fixed much of that by splitting each batch into micro-batches, so different stages can process different micro-batches at once.
  • Within a layer: tensor parallelism. The big matrix multiplications inside each layer are split, so every device computes a slice of every layer. Megatron-LM showed this can be done for transformers with a few added communication operations. Because devices must sync inside every layer, it works best over very fast links, typically between GPUs in the same server.

A related idea routes different inputs to different parts of the model: mixture-of-experts models place experts on different devices, a split usually called expert parallelism.

The catch

Splitting a model turns memory problems into communication problems. Pipelines leave some devices idle at the start and end of each batch; tensor splits add traffic to every layer. That’s why large training runs mix model parallelism with data parallelism and sharding, choosing the split that suits each level of the hardware.

Where it sits

Before this

Training GPU Layer
Model Parallelism

Leads to

Pipeline Parallelism Tensor Parallelism

Explore nearby

Training Tensor Parallelism Splitting individual layers/tensors across devices for very large models. Training Pipeline Parallelism Splitting model layers across devices and processing micro-batches in pipeline fashion. Training Data Parallelism Replicating the model across devices, each processing different data batches. Training Sharded Data Parallelism Distributing model states across devices to train models larger than single-device memory. Neural Networks Mixture of Experts An architecture where multiple specialized sub-networks (experts) process inputs, with a gating network routing to relevant experts.

Sources

  1. Yanping Huang et al., "GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism" . NeurIPS 2019
  2. Mohammad Shoeybi et al., "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism" . arXiv, 2019; intra-layer (tensor) model parallelism