Measuring whether it works.
THE BASIC APPROACH
Hold back a portion of the data. Train on the rest. Measure performance on the held-back portion.
That measures performance on unseen data, which is what matters.
ACCURACY
The proportion of correct predictions. Simple and frequently misleading.
WHY ACCURACY MISLEADS
If one per cent of transactions are fraudulent, a model predicting no fraud ever is ninety-nine per cent accurate and useless.
BETTER MEASURES
Precision: of the items flagged, how many were correct
Recall: of the items that should have been flagged, how many were
THE TRADE-OFF
Increasing one usually decreases the other.
A fraud system catching everything also flags many legitimate transactions.
WHICH MATTERS DEPENDS ON THE COST
Missing a fraudulent transaction versus annoying a legitimate customer.
Missing a disease versus an unnecessary test.
That is a business decision, not a technical one.
WHAT TO ASK A VENDOR
Not "how accurate is it" but "what does it miss, and what does it flag wrongly".
