Standard 4 stops to get here
A/B Testing
Comparing two model versions in production by routing traffic to each and measuring performance differences.
Your route here
4 stops · basics first
- Dataset ✓ understood
A collection of data examples used for training, validating, or testing machine learning models.
- Training ✓ understood
The process of fitting a model to data by repeatedly measuring how wrong its outputs are and adjusting its parameters to reduce that error.
- Inference ✓ understood
Running a trained model on new inputs to get predictions, with its weights frozen: the stage of a model's life that users actually interact with.
- Model Serving ✓ understood
Deploying trained models as services that can handle prediction requests in production environments.
- A/B Testing · you are here ✓ understood
Where it sits
Explore nearby
Agents & RL Multi-Armed Bandit The simplest decision problem with a trade-off between exploring and exploiting: repeatedly pick one of several options with unknown payouts to maximize total reward. Evaluation Type I and Type II Errors The two ways a yes-or-no decision can be wrong: a Type I error is a false alarm (false positive), a Type II error is a miss (false negative). Shipping AI Shadow Deployment Running a new model alongside production without affecting user experience, for validation. Shipping AI Canary Deployment Gradually rolling out new model versions to subset of traffic before full deployment. Shipping AI Model Monitoring Tracking model performance, data distribution, and predictions in production to detect issues and degradation. Shipping AI Experiment Tracking Recording hyperparameters, metrics, and artifacts from training runs for comparison and reproducibility. Evaluation Baseline Model A simple reference model (random, majority class, simple heuristic) used to benchmark more complex models against.