Data annotation is the process of attaching the answers to raw data: marking an email as spam, drawing a bounding box around each car in a photo, transcribing an audio clip, or ranking two chatbot replies. The output is labeled data, the fuel for supervised learning and the yardstick for evaluation. Most annotation is still done by people, following written guidelines, often through crowdsourcing platforms or specialist vendors.
Quality is the whole game
Annotators get tired, misread instructions, and genuinely disagree, especially on subjective calls like sentiment or toxicity. Standard practice is to have several people label the same items, measure how often they agree with a statistic like Cohen’s kappa, and combine their answers. Snow and colleagues showed in 2008 that cheap non-expert labels collected on Amazon’s Mechanical Turk agreed closely with expert gold-standard labels on five language tasks, at a fraction of the cost.
Even careful pipelines leave errors. Northcutt and colleagues estimated that at least 6% of the ImageNet validation labels are wrong. Your ground truth is only as true as the people and the guidelines that produced it.
Cutting the cost
Annotation is slow and expensive, so much of machine learning is about needing less of it. Active learning asks people to label only the examples the model is most unsure about. Semi-supervised and self-supervised methods learn from unlabeled data, and models can pre-label data for humans to correct.
Annotation for language models
Modern LLMs depend on annotation too. InstructGPT, an influential recipe for instruction-following models, was trained on demonstrations written by human labelers and on labelers’ rankings of model outputs, the preference data at the heart of RLHF.