Choosing what becomes a test.
WHAT AUTOMATION PROVIDES
Repeatability Speed Execution on every change Freedom from human error in repetition
WHAT IT DOES NOT PROVIDE
Judgement Discovery of unanticipated problems Assessment of whether something looks right
WHAT TO AUTOMATE
Tests run frequently Tests verifying stable behaviour Tests too tedious to run reliably by hand Checks of things easily broken
WHAT NOT TO AUTOMATE
Behaviour still changing frequently One-off verification Things requiring visual or experiential judgement Areas where the cost of maintaining the test exceeds the value
WHAT DETERMINES THE VALUE
Times the test will run, multiplied by the cost of the defect it catches.
WHAT DETERMINES THE COST
Writing it, plus maintaining it every time the software changes.
WHY MAINTENANCE DOMINATES
Tests break when behaviour changes legitimately, and each break requires attention.
WHAT THAT ARGUES AGAINST
Automating everything.
WHAT THE RIGHT PROPORTION IS
Enough to give confidence to release, and no more.
WHAT TO REVIEW
Tests that fail frequently without finding defects.
WHAT TO DO WITH THEM
Fix or delete them.
