Measuring generated output.
WHY IT IS DIFFICULT
There is no single correct output, and quality is multidimensional.
WHAT TO BUILD FIRST
An evaluation set of representative inputs with expected characteristics.
WHY BEFORE ANYTHING ELSE
Without it, every change is judged by impression.
WHAT TO MEASURE
Correctness, where verifiable Relevance to the question Faithfulness to supplied sources Format compliance Refusal behaviour, where appropriate Latency and cost
WHAT FAITHFULNESS MEANS
Claims in the output supported by the supplied material.
WHY IT IS THE KEY METRIC FOR RETRIEVAL SYSTEMS
Output that sounds right but is unsupported is the primary failure mode.
WHAT AUTOMATED EVALUATION USES
Exact or fuzzy matching, where answers are constrained Another model as a judge Rule-based checks for format and forbidden content
WHAT TO BE CAREFUL WITH IN MODEL-BASED JUDGING
Bias toward verbose answers Inconsistency between runs Cost
WHAT TO COMBINE IT WITH
Human review of a sample.
WHAT TO RUN EVALUATION ON
Every change: prompt, model, retrieval, parameters.
WHAT TO TRACK OVER TIME
Whether quality is improving or regressing.
