Model Serving with A/B Testing
Route inference traffic across model versions to compare live performance.
IntermediateAI / MLPlatform
An inference gateway splits traffic between a champion and a challenger model version, logging outcomes for each so the challenger's real-world performance can be measured against the champion before it fully replaces it.
When to use it
- A new model version needs live validation before becoming the default
- Offline evaluation metrics don't fully capture real-world outcome quality
Trade-offs
- Needs an outcome-labelling pipeline to actually compare champion vs challenger
- Running two model versions concurrently doubles serving cost for the test duration
Components used
Single-Page AppAI GatewayLLM / Model EndpointModel RegistryMetrics Store
How it works
- A routing layer sits in front of two or more model versions and splits inference traffic by a stable hash of the user or session id, so a given user always sees the same variant.
- Every prediction is logged with its variant tag alongside the downstream business outcome, which is what actually gets compared.
- Offline evaluation metrics and live business metrics regularly disagree; this pattern exists because only the live comparison settles the argument.
Used in the wild
- Validating that a retrained recommendation model actually lifts engagement before full rollout.
- Comparing a cheaper distilled model against the incumbent to see whether quality loss is noticeable.
- Shadow-testing a new fraud model against the production one on identical traffic.
Good to know
- Hash on a stable id, never randomly per request. Random assignment lets one user flip between models mid-session, which both ruins the experiment and produces visibly inconsistent behaviour.
- Run an A/A test first — route traffic to two copies of the same model. If it reports a significant difference, your measurement pipeline is broken, not your model.
Related patterns
Retrieval-Augmented Generation (RAG)
Ground an LLM's answers in retrieved, up-to-date, private documents.
Agentic Tool-Use Loop
An agent plans, calls tools, observes results, and iterates to a goal.
Vector Search + Rerank
Cheaply retrieve a broad candidate set, then precisely re-rank the top results.
Fine-Tuning Pipeline
Curate a training set, fine-tune a base model, and register the result.