Information retrieval (IR) is the science behind search. The standard textbook definition, from Manning, Raghavan and Schütze, is finding material, usually text documents, that satisfies an information need from within a large collection. Web search engines are the best-known example, but the same machinery runs inside email search, legal discovery, code search and, increasingly, the retrieval step of retrieval-augmented generation.
How it works
An IR system prepares an index ahead of time so it never has to read the whole collection per query. Classic systems build an inverted index, a map from each word to the documents containing it. At query time the system pulls candidate documents and ranks them by estimated relevance.
- Sparse (lexical) retrieval scores documents by the words they share with the query, weighted so rare, telling words count more. TF-IDF and its refinement BM25 have been the workhorses for decades.
- Dense retrieval encodes queries and documents as embeddings and ranks by vector similarity, which finds matches that share meaning but not words. This is what people usually mean by semantic search, and it is typically served from a vector database.
Karpukhin et al. found that the two fail differently: term matching is sensitive to highly selective keywords, while dense retrieval better captures related wording. Many production systems combine both, then re-rank the top results with a slower, more accurate model.
Measuring it
Retrieval quality is measured with ranked-list metrics built from precision and recall: of what came back, how much was relevant, and of what was relevant, how much came back, with extra credit for putting the best results first.