Landmark 2 stops to get here · leads to 3

Parameter

A number inside a model, a weight or a bias, that training adjusts to reduce the loss. Together, a model's parameters are everything it has learned.

Your route here

2 stops · basics first
  1. Machine Learning ✓ understood

    Building systems that learn patterns from data instead of following hand-written rules, getting better at a task as they see more examples.

  2. Neural Network ✓ understood

    A model built from layers of simple units, each taking a weighted sum of its inputs and applying a nonlinearity; training adjusts the weights so the whole stack maps inputs to useful outputs.

  3. Parameter · you are here ✓ understood

Picture it

Parameters

  • Weights and biases inside the model
  • Learned from data during training
  • Billions in a large model
  • Updated by gradient descent

Hyperparameters

  • Settings chosen before training
  • e.g. learning rate, batch size, layers
  • Set by you or a tuning search
  • Not updated by gradient descent
Notice the split: parameters are what the model learns, hyperparameters are the knobs you set to control how it learns.

A parameter is one of the adjustable numbers inside a model. In a neural network they come in two kinds: weights, which set how strongly one unit’s output feeds into the next, and biases, which shift each unit’s output up or down. Training starts them at random values and nudges every one of them, step after step, to make the loss smaller.

When the run ends, the parameters are the model. The architecture is only a shape; the learned numbers are what make it translate French or spot tumours. Downloading “a model” means downloading its parameters.

Counting them

A fully connected layer with 1,000 inputs and 1,000 outputs has a weight for every input-output pair plus a bias per output: 1,000,000 + 1,000 parameters. Stack enough layers and the total climbs fast. GPT-3, in 2020, had 175 billion. That count is the “175B” in a model name, and it’s the most common shorthand for model size.

Counts are less standard than they look. A 2020 study of scaling laws deliberately left out the embedding table, because the trend was cleaner that way. Mixture-of-experts models have a total count and a much smaller count of parameters actually used for each token.

Why the number matters

Parameter count drives cost in two places.

  • Memory: every parameter has to be stored. At 16 bits each, 175 billion parameters take about 350 GB before you’ve processed anything. Quantization stores them in fewer bits to shrink that.
  • Compute: a forward pass through a transformer costs roughly two operations per parameter for each token processed, so bigger models are slower and pricier to run.

More parameters give a model more capacity to represent complicated patterns, and within limits loss falls predictably as size grows.

The catch

Parameters alone don’t make a model good. The Chinchilla study in 2022 found that many large language models were significantly undertrained for their size; a 70-billion-parameter model trained on more data beat much larger ones, GPT-3 included. Data, training compute and parameters have to grow together.

Don’t confuse parameters with hyperparameters. The diagram draws that line: parameters are learned by gradient descent, hyperparameters are the settings you choose before training starts.

Where it sits

Explore nearby

Sources

  1. Brown et al., "Language Models are Few-Shot Learners" . 2020 (GPT-3, 175 billion parameters)
  2. Kaplan et al., "Scaling Laws for Neural Language Models" . 2020; section 2.1, parameter and compute counts
  3. Hoffmann et al., "Training Compute-Optimal Large Language Models" . 2022 (Chinchilla)