Fine-tuning is taking a model that has already been trained and training it a little more on a smaller, focused dataset, so it gets good at a particular task, domain or style. The expensive general learning happened during pre-training; fine-tuning adjusts the result rather than starting over.
The payoff is efficiency. Because the model already knows a lot, it needs far fewer examples to learn something new. In the 2018 ULMFiT paper, a text classifier fine-tuned from a pre-trained language model on just 100 labelled examples matched one trained from scratch on 100 times as much data.
How it works
The diagram shows the loop. Start from the pre-trained weights, prepare a task-specific dataset, and continue training with ordinary gradient descent. Two details matter. The learning rate is usually set lower than in pre-training, so the updates refine the model instead of overwriting it. And you evaluate on a held-out set throughout, watching both for gains on the new task and for losses on what the model could already do.
The main kinds
- Full fine-tuning updates every weight. It is the most flexible, but you end up storing a complete copy of the model for each task.
- Parameter-efficient fine-tuning freezes the original weights and trains a small add-on. A widely used method, LoRA (2021), learns small low-rank matrices beside each weight layer. On GPT-3, it cut the trainable parameters by a factor of 10,000 and GPU memory use by a factor of three, while matching full fine-tuning on quality.
- Instruction tuning fine-tunes on examples of requests paired with good responses, which turns a base model that only continues text into one that follows instructions.
- Preference tuning, such as RLHF, goes a step further and trains on human judgments of which answer is better. OpenAI’s InstructGPT combined both steps, and people preferred answers from its 1.3-billion-parameter model to those from the original 175-billion-parameter GPT-3.
The catch
Training hard on a narrow dataset can make a model lose abilities it had before, a problem called catastrophic forgetting. Lower learning rates, fewer training steps and mixing some general data back in all reduce the risk.
Fine-tuning is also not always the right tool. It is good at teaching a format, a style or a skill. For supplying facts that change, retrieving them or putting them in the prompt is often simpler, and easier to keep current than retraining.