Vector Search + Rerank
Cheaply retrieve a broad candidate set, then precisely re-rank the top results.
IntermediateAI / MLData
A fast approximate-nearest-neighbour vector search pulls a wide candidate set, then a more expensive cross-encoder reranker reorders just those candidates for precision -- balancing recall and latency instead of picking one extreme.
When to use it
- Pure vector similarity search returns plausible but imprecisely ordered results
- You can afford a heavier model on a small candidate set but not on the full corpus
Trade-offs
- Two models to run and keep in sync (embedding model and reranker)
- Reranking adds latency proportional to the candidate set size
Components used
Managed App ServiceEmbedding ServiceVector DatabaseLLM / Model Endpoint
How it works
- A bi-encoder embeds queries and documents independently, so document vectors can be precomputed and searched with fast approximate nearest-neighbour lookup. This retrieves maybe 100 candidates cheaply.
- A cross-encoder then scores each (query, document) pair jointly. It is far more accurate but must run at query time, which is why it only ever sees the shortlist.
- The two-stage split is a deliberate cost/accuracy trade: broad and cheap first, narrow and precise second.
Used in the wild
- Improving RAG answer quality without changing the LLM or the chunking strategy.
- Product search where semantic recall matters but the top three results must be exactly right.
- Deduplicating near-identical records where embedding similarity alone produces too many false positives.
Good to know
- Reranking is usually the highest-leverage fix for a mediocre RAG system. Teams reach for a bigger model first, when swapping in a cross-encoder reranker typically moves the needle more for a fraction of the cost.
- Cross-encoders are roughly two orders of magnitude slower per pair than bi-encoders, which is precisely why nobody runs them over the whole corpus.
Related patterns
Retrieval-Augmented Generation (RAG)
Ground an LLM's answers in retrieved, up-to-date, private documents.
Agentic Tool-Use Loop
An agent plans, calls tools, observes results, and iterates to a goal.
Model Serving with A/B Testing
Route inference traffic across model versions to compare live performance.
Fine-Tuning Pipeline
Curate a training set, fine-tune a base model, and register the result.