Measuring whether an AI quality system could be trusted
Built a 20,000-line evaluation harness for a GMP AI system: top-20 retrieval went from about 3-in-10 to about 8-in-10, and the unsafe recommendations I'd seen went from 3 to 0 across 201 test cases.
The problem. Cellares was building an AI-assisted quality management system to help investigators handle manufacturing deviations. Part of it searched a library of controlled procedures (SOPs) to find the ones that applied. In a GMP environment you can't ship that on a hunch, and nobody could say how well it actually worked.
What I did. I built an evaluation harness from scratch, about 20,000 lines of Python. It uses golden datasets built from real historical deviations, reviewed by subject-matter experts and de-identified. It scores retrieval with recall, hit rate, MRR and NDCG, scores answers with an LLM-as-judge using a GMP-specific rubric, and runs as a CI quality gate that fails closed if it checked nothing. The system under test sits behind one interface, so that a configuration change shows up as a measured difference instead of a rebuild. Embeddings and reranking ran inside our environment, so no regulated content went to an outside evaluation vendor.
What happened.
- Landing the right procedure somewhere in the top 20 results went from about 3 in 10 to about 8 in 10 (recall@20 ≈ 0.78, hit rate ≈ 0.79 on 86 cases over 499 documents).
- The AI's recommendations to close or cancel a real quality event went from 3 to 0 across 201 test cases, though the honest version of that is a story about test data more than about the model. Two of the three came from a defect in my own de-identification that had rewritten a deviation's own record number into a different one, so the case read as a duplicate of an event that already existed and the model reasoned correctly to the wrong conclusion. Fixing the data removed them. With counts that small this is really "none observed" rather than proof of zero, since the 95% upper bound is still around 1.5%, but it's the failure mode with regulatory consequences, so it's the one I watch hardest and the one I keep expanding the test set for.
- The harness also turned up problems that had nothing to do with the AI. 64 of 467 in-force documents, about 14%, were invisible to search, and five platform defects got caught before users saw them.
- The whole ten-experiment campaign cost under $5 in embeddings.
What I'd tell someone else. Build the measurement before the features, and then measure the measurement. I stopped trusting any difference smaller than about 0.08 in either direction, which is roughly the run-to-run wobble and also about the width of the confidence interval you get from only 86 test cases, so the honest way to report these numbers is two figures rather than three, and the way to tighten them is more cases rather than more re-runs. That same habit caught a hard-coded result limit that was quietly making good results look missing, before it cost me a full re-embed of the corpus.