The summary.
QUERY TERMS MUST BE ANALYSED IDENTICALLY TO INDEXED TERMS
Otherwise nothing matches, and the cause is rarely obvious.
Start with the database's own search — it needs no extra infrastructure and suffices for many applications.
SEARCH LOGS TELL YOU WHAT IS MISSING FROM YOUR PRODUCT
Queries returning nothing, and queries where nothing is clicked, are the most valuable data you have.
OPTIMISING ENGAGEMENT ALONE PRODUCES SYSTEMS PEOPLE USE COMPULSIVELY AND RESENT
Measure downstream outcomes and diversity too. Clicks are easy to increase and frequently mean nothing.
Popularity bias narrows what everyone sees and prevents new items surfacing — deliberate exploration is the remedy.
OFFLINE EVALUATION CANNOT SEE HOW USERS WOULD RESPOND TO WHAT THEY NEVER SAW
That feedback loop bias is pervasive in recommendation work.
BECAUSE FRAUD IS RARE, A SMALL FALSE POSITIVE RATE REJECTS MORE LEGITIMATE CUSTOMERS THAN FRAUDSTERS
Measure the false positive rate explicitly, and always provide a review path — otherwise you lose those customers permanently.
ALERT FATIGUE IS THE PRINCIPAL FAILURE MODE OF DETECTION SYSTEMS
Tune to the rate humans can actually investigate.
