A parameter is one of the adjustable numbers inside a model. In a neural network they come in two kinds: weights, which set how strongly one unit’s output feeds into the next, and biases, which shift each unit’s output up or down. Training starts them at random values and nudges every one of them, step after step, to make the loss smaller.
When the run ends, the parameters are the model. The architecture is only a shape; the learned numbers are what make it translate French or spot tumours. Downloading “a model” means downloading its parameters.
Counting them
A fully connected layer with 1,000 inputs and 1,000 outputs has a weight for every input-output pair plus a bias per output: 1,000,000 + 1,000 parameters. Stack enough layers and the total climbs fast. GPT-3, in 2020, had 175 billion. That count is the “175B” in a model name, and it’s the most common shorthand for model size.
Counts are less standard than they look. A 2020 study of scaling laws deliberately left out the embedding table, because the trend was cleaner that way. Mixture-of-experts models have a total count and a much smaller count of parameters actually used for each token.
Why the number matters
Parameter count drives cost in two places.
- Memory: every parameter has to be stored. At 16 bits each, 175 billion parameters take about 350 GB before you’ve processed anything. Quantization stores them in fewer bits to shrink that.
- Compute: a forward pass through a transformer costs roughly two operations per parameter for each token processed, so bigger models are slower and pricier to run.
More parameters give a model more capacity to represent complicated patterns, and within limits loss falls predictably as size grows.
The catch
Parameters alone don’t make a model good. The Chinchilla study in 2022 found that many large language models were significantly undertrained for their size; a 70-billion-parameter model trained on more data beat much larger ones, GPT-3 included. Data, training compute and parameters have to grow together.
Don’t confuse parameters with hyperparameters. The diagram draws that line: parameters are learned by gradient descent, hyperparameters are the settings you choose before training starts.