Prompts as production code: auditing 49 templates across 78 endpoints

Context

At STAND 8 I owned PRISM's prompt orchestration layer: prompts lived as versioned YAML templates and were composed at request time for each endpoint. Over months of fast iteration the layer grew to 49 templates serving 78 production endpoints — which is the point where "it works on my test case" stops being good enough.

The audit

I led a full system audit of the layer. Two classes of bug stood out:

• prompt_id collisions — different templates resolving to the same identifier, so the wrong prompt could be served silently.
• A broken composer config — template fragments assembled incorrectly for some endpoints.

Neither throws an error. Both just make outputs a little worse, which is the hardest kind of failure to notice in an LLM system.

What we built so it doesn't happen again

• A regression harness tied to the golden sets from the evaluation work.
• Dashboards for token usage, prompt latency, and per-stage failure rates.
• Automated A/B prompt evaluation, which cut per-candidate cost.
• A structured-extraction service writing 60+ fields into PostgreSQL JSONB with a versioned schema and dynamic field exclusion, so we only pay tokens for fields a workflow needs.

I also wrote stakeholder-ready explainers of the system for team leads — research that nobody else can read doesn't change decisions.