Before changing prompts or models, write down what a good result means. A small, representative evaluation set is more useful than dozens of polished screenshots.
Build cases from real work
Collect normal requests, difficult edge cases, ambiguous inputs, and hostile content. Remove sensitive data and preserve the structure of the task.
Score the dimensions separately
- Task completion and factual support
- Safety and policy compliance
- Latency and cost
- Clarity for the person who must use the result
Keep human review in the loop
Automated graders help with scale, but a recurring human review catches subtle regressions. Record disagreements and turn them into better rubrics or new test cases.
Ship with a rollback threshold
Decide in advance which failure rate, latency increase, or cost spike pauses the rollout. Evaluations are most valuable when they drive an operational decision.