Standardised model testing.
THE DEFINITION
Standardised tests measuring model performance on defined tasks.
WHAT THEY ARE USEFUL FOR
Rough comparison between models Tracking progress over time
WHAT THEY ARE NOT USEFUL FOR
Predicting performance on your specific task.
THE CONTAMINATION PROBLEM
Benchmark questions appear on the internet, which is where models learn.
A model may have seen the test, which inflates scores without indicating capability.
WHAT VENDORS DO
Report benchmarks favourable to their model. Every model is best at something.
WHAT TO DO INSTEAD
Build a small set of representative tasks from your own work and compare on those.
Twenty examples from your actual work tells you more than any published score.
RELATED TERMS
Model evaluation, accuracy, precision and recall.
