Two textbook RAG improvements that didn't earn their keep
Cross-encoder reranking and hybrid search are the standard next steps for retrieval. In the configs I tried, one scored worse and was far too slow, the other showed no measurable difference, and the wins came from boring changes.
The problem. Cross-encoder reranking and hybrid keyword-plus-vector search are the standard next steps for improving retrieval, so I expected at least one of them to help.
What I did. I implemented both and measured each one in isolation against the same dataset, at the same cutoff of 20 results.
What happened. Reranking scored worse. In the pool the shipped pipeline actually uses, one best passage per document, adding the cross-encoder took recall from about 0.57 to 0.43 (hit rate about 0.60 to 0.47) at the top 10 across 87 real-deviation cases, and a production-shaped stage over a deeper pool still regressed every retrieval metric, so the drop held on a second and independent path. It also ran at about 163 seconds per query on CPU, which made it a non-starter for us regardless of accuracy, so I didn't invest in debugging or tuning it, and a faster or better-tuned reranker might land differently, but at that latency it wasn't worth finding out. Hybrid search with reciprocal rank fusion showed no measurable difference from vector-only, so I dropped that one too. The wins came from less glamorous changes. Returning one best passage per document instead of several near-duplicate chunks took recall from about 0.45 to about 0.57, widening the cutoff from 10 to 20 results was a deliberate trade of precision for recall, and chunking by section with the headings kept in was the third.
What I'd tell someone else. The obvious improvement is a hypothesis, not a plan. Five of our ten experiments were rejections, and those were worth as much as the wins. The flip side is knowing when not to chase one, and a reranker that was both slower and worse didn't earn a debugging session when the latency alone ruled it out.