Testing changes deliberately.
WHAT AN EXPERIMENT IS
A change made to test a specific expectation.
WHAT TO DEFINE BEFORE RUNNING IT
What you expect to happen What you will measure How long it runs What result would change your decision
WHY IN ADVANCE
Otherwise results are interpreted to support what you already wanted.
WHAT COMPARISON TESTING DOES
Shows two versions to different groups, and compares outcomes.
WHAT IT REQUIRES
Enough users for the difference to be meaningful A measure that reflects the real outcome Discipline not to stop early
WHY NOT STOPPING EARLY MATTERS
Early results fluctuate, and stopping at a favourable moment produces false conclusions.
WHAT SMALL BUSINESSES CAN DO INSTEAD
Sequential testing: change something, measure, change back if worse.
WHAT ITS LIMITATION IS
Other factors change at the same time.
WHAT TO DO ABOUT THAT
Test one thing at a time, and be cautious about conclusions.
WHAT TO TEST
Changes you genuinely disagree about Changes with substantial consequences
WHAT NOT TO TEST
Obvious improvements Things you would implement regardless
WHY
Testing has a cost, and it should resolve real uncertainty.
WHAT TO RECORD
The expectation, the result and the decision.
WHY
It builds understanding of your users over time.
WHAT TO ACCEPT
That most changes produce smaller effects than hoped.
