Vibes are a terrible release gate
The prototype feels good because you know the happy path. Production finds the weird path immediately: short prompts, angry prompts, copied tickets, partial context, old product names, and users who ask three things at once. That is why an eval set is not academic overhead. It is a tripwire.
A small replay beats a confident demo.
Hit the button and watch the hidden AI surface area light up.
Start with the failures you fear
A useful first eval set is not huge. It is honest. Include common requests, edge cases, policy boundaries, prompt injection attempts, and examples from real support tickets after removing private data. Each case should say what good looks like in plain language.
Readable beats clever. If a product manager can review the case, you are on the right track.
Score behavior, not eloquence
Pretty answers can still be wrong. The rubric should check concrete behavior: cites the right source, refuses the forbidden task, asks the missing question, routes the risky topic, or returns structured JSON. Let style be a smaller score. Accuracy gets the veto.
Do the boring checks first: exact match, schema match, source match, refusal match. Bring in an LLM judge only when rules cannot express the behavior.stdout rule of thumb
Flip the switches and watch the story turn into a tiny operating model.
Run evals whenever the system changes
The model changed? Run evals. The prompt changed? Run evals. The docs changed? Run evals. The retrieval code changed? Definitely run evals. You are building a moving system, so quality has to move with it.