Reranking

Reranking is a second-pass scoring step that takes the candidate documents returned by an initial retrieval and reorders them with a more expensive model before they reach the language model. The first stage optimizes for recall, getting the right passage somewhere into the top fifty; the reranker optimizes for precision, deciding which five of those fifty are worth the context budget.

The split exists because the two jobs need different architectures. Vector search uses a bi-encoder: query and document are embedded separately, so document vectors are computed once at index time and compared with a cheap similarity operation at query time. That independence is what makes the index fast, and it is also the limitation, because the model never sees the query and the document together. A cross-encoder runs both through a single forward pass and can score how well this document answers this query, which is more accurate and far too slow to run across a million documents. Running the cheap model over everything and the expensive model over the survivors is the standard resolution. LLM-based rerankers extend the same idea by prompting a general model to order the candidates, at higher quality and higher latency.

For anyone building on a hosted search API, the important thing is that reranking may already have happened. AI search APIs are sold as a single call that returns ranked, extracted content, and the ordering inside is not raw vector similarity. Exa combines neural retrieval with its own ranking layer, Tavily and Linkup score and filter results before returning them, and Parallel AI is built around multi-hop research where intermediate ranking is the product rather than a stage you configure. Reranking that output again is worth it mainly when your relevance criteria are domain-specific in a way a general model cannot infer: a legal product cares about jurisdiction, a code assistant cares about language and version, and neither preference is visible in the query text.

The cost shows up as latency. A hosted rerank call adds a network round trip plus scoring time proportional to the number and length of candidates, so reranking 50 short passages and reranking 500 long ones are different line items. Check the vendor's own published latency figures at your candidate count rather than the headline number. Inside an agent that already spends a second on search and several on generation, that is invisible. Inside an autocomplete-style feature it is the entire budget. Vendor quality claims should be read as vendor claims: Voyage AI's August 2025 announcement of rerank-2.5 reported single-digit percentage NDCG gains over Cohere Rerank v3.5 across its own evaluation set, which is a reasonable signal about direction and a weak one about your corpus.

Checking whether reranking earns its place takes an afternoon. Log a few hundred real queries, record what first-stage retrieval returned, rerank them offline, and check two numbers: whether the correct passage moved into the top k more often, and whether recall fell because the reranker demoted something the generator needed. Rerankers routinely win precision by discarding a supporting document that a multi-hop question required, and that failure stays invisible unless you measure recall before and after rather than only the top-1 hit rate.