Practical evaluation.
BUILD A TEST SET
Ten to twenty representative tasks from your actual work.
Include easy ones, difficult ones, and edge cases.
INCLUDE TASKS WHERE YOU KNOW THE ANSWER
That is how you detect errors.
WHAT TO ASSESS
Accuracy Whether output needs heavy editing Consistency across attempts How it handles being uncertain Speed
THAT FOURTH POINT
Ask something it cannot know. Does it say so, or invent an answer?
That tells you a great deal.
WHAT TO TRY
The same task several times, to see variability Deliberately ambiguous requests Requests outside its stated purpose
COMPARE PROPERLY
Same tasks, same prompts, different tools.
WHAT NOT TO DO
Judge from a demonstration Judge from one impressive result Judge from marketing
HOW LONG TO TEST
Long enough to see it fail. If you have not seen it fail, you have not tested it.
