Reranking is a second scoring pass in a retrieval pipeline. A fast first-stage retriever (vector search, keyword search such as BM25, or a hybrid of both) pulls back a broad set of candidate documents, and then a more accurate model called a reranker reads the user’s query alongside each candidate, assigns a relevance score, and reorders the list. Only the top few results are passed to the LLM. In retrieval-augmented generation (RAG), adding a reranker is often the single change that turns “the chunks look related but the answer is wrong” into answers grounded in the right source.

Why it matters now: as companies move RAG assistants and AI agents from demos into production, retrieval quality has become the bottleneck rather than model size. Pinecone’s guide to rerankers and two-stage retrieval explains why search engineers have relied on this pattern for years, and vendors and frameworks now ship reranking as a built-in step. If you are new to how semantic search works, start with our explainer on embeddings and why RAG depends on them; reranking is the layer that fixes what embeddings alone get slightly wrong.

Why One Retrieval Step Is Not Enough

Most RAG systems start with a bi-encoder: an embedding model that turns every document chunk into a vector ahead of time and turns the query into a vector at request time. Similarity search over those vectors is extremely fast, even across millions of chunks, which is why it powers vector databases.

The trade-off is information loss. Each document is compressed into a single vector before anyone knows what question will be asked, so the representation is a generic average of its meaning. Two chunks can sit close together in vector space while only one of them actually answers the question. Negations, exact product names, numbers, and subtle qualifiers are where bi-encoders most often slip.

How a Cross-Encoder Reranker Works

A cross-encoder takes the query and one candidate document together as a single input, runs them through a transformer with full attention across both texts, and outputs one relevance score. Because the model reads the pair jointly, it can notice that the document mentions the right feature but for the wrong product, or that it answers a slightly different question. The downside is cost: nothing can be precomputed, and every query–document pair needs its own forward pass. That makes cross-encoders too slow to scan a whole corpus, but perfectly practical for re-scoring a few dozen candidates.

The Two-Stage Retrieval Pattern

• Stage 1: retrieve for recall. Use vector, keyword, or hybrid search to pull a broad candidate pool, commonly 50 to 100 chunks. The goal is simply to get the right document somewhere in the set.

• Stage 2: rerank for precision. Score every candidate with the reranker and sort by that score.

• Stage 3: keep the best few. Pass the top 5 to 10 chunks into the LLM’s context window and discard the rest before they cost you tokens.

As the practitioner guide Reranking in RAG with Cross-Encoders summarizes it: retrieve broadly and cheaply, then rerank narrowly and precisely. One rule follows directly from this design: a reranker can only reorder what it is given. If the right document never made it into the candidate pool, no reranker can rescue it, so tune first-stage depth before blaming the reranker.

Types of Rerankers

Cross-encoders are the most common choice, available as open models (for example MiniLM- or BGE-based rerankers) and as hosted APIs from search and LLM vendors. Late-interaction models such as ColBERT store a vector per token at ingest time and compare query tokens to document tokens at search time, trading more storage for lower latency. LLM-based rerankers ask a general-purpose language model to compare a short list of candidates at once (list-wise reranking); they can be very accurate but are the slowest and most expensive option, so teams usually reserve them for small, high-value candidate sets.

When a Reranker Is Worth It

Reranking adds latency, typically tens to a few hundred milliseconds depending on the model, hardware, and number of candidates. It pays off when answers depend on precise details, when your corpus has many near-duplicate or similar-looking chunks, or when users ask long, specific questions. The Qdrant guide on when a reranker is worth it recommends measuring it like any other change: establish a baseline ranking, score the same candidates with and without the reranker using a metric such as nDCG@10, and keep it only if quality improves enough to justify the extra latency.

How to Measure the Impact

Build a small evaluation set of real questions with known relevant documents. Track retrieval metrics (recall@k, nDCG, MRR) and end-to-end answer quality, plus p95 latency and cost per query. Our guide to AI evals covers how teams turn these checks into repeatable tests, so a model swap or chunking change does not silently degrade results.

Common Mistakes With Reranking

Retrieving too few candidates, so the reranker has nothing better to promote. Reranking hundreds of candidates and blowing the latency budget. Choosing a reranker trained only on English for a multilingual corpus. Passing every reranked chunk to the LLM instead of cutting to the top few. And skipping evaluation, assuming the reranker helped because the pipeline “feels” smarter.

If you want to build RAG systems and AI agents that retrieve the right information and prove it with data, explore Coderhouse’s courses below.

• AI Engineering Course — design production RAG pipelines with retrieval, reranking, and evaluation.

• Data Analytics Course — learn the metrics and analysis skills to measure retrieval quality and cost.

FAQ

Is reranking the same as re-embedding my documents?

No. Re-embedding changes the vectors used for first-stage search. Reranking keeps those vectors and adds a second model that re-scores the retrieved candidates at query time.

Can I use a reranker without a vector database?

Yes. Rerankers work on top of any first-stage retrieval, including classic keyword search. Many teams add a reranker to an existing BM25 search before adopting embeddings at all.

How many documents should I rerank?

A common starting point is retrieving 50 candidates and keeping the top 5 to 10. Increase the candidate pool until recall stops improving, then pick the final cut based on answer quality and latency.