When the labels lie: auditing an LLM evaluation pipeline

The question

PRISM is an AI recruiting platform I led at STAND 8 (2025–26): it ingests resumes, recruiter notes and voice-screening transcripts, then extracts, enriches and ranks candidates. Every week someone wanted to change a prompt, swap a model, or trim a field to save tokens. The research question underneath all of that was simple to state and annoying to answer: did this change actually make the system better?

Setup

I built the evaluation framework as a first-class part of the product rather than a notebook on the side:

• Offline golden test sets — fixed, versioned inputs with expected outputs, per pipeline stage.
• An automated regression harness that re-runs those sets on every prompt or model change.
• LLM-as-judge scoring for outputs where exact match is meaningless (summaries, rationales).
• Dashboards for structured human review, so disagreements between the judge and a person become data instead of anecdotes.

What it found

Two results mattered more than any single benchmark number.

First, a 43-point accuracy gap between models on the same extraction task. Both models looked fine in spot checks; the gap only appeared once they were scored side by side on the same golden set. That finding directly shaped the multi-LLM routing layer, which dispatched GPT-4o and Gemini by workload type (extraction, reasoning, summarisation) instead of using one model everywhere.

Second — and more uncomfortable — a batch of the ground-truth labels were themselves machine-generated and invalid. They had been quietly skewing results: some "regressions" were the model being right and the label being wrong.

An evaluation is only as honest as its labels. Before trusting a delta, trace where every expected answer came from.

What I'd tell my past self

1. Track label provenance (human, model, or heuristic) as a column, not a memory.
2. Treat the eval set as code: versioned, reviewed, and diffed like everything else.
3. Route judge–human disagreements to review first; that is where the bugs hide.

Open questions

How well-calibrated is an LLM judge across model families it is scoring? How small can a golden set be before it stops catching real regressions? These are the questions I keep returning to.