Measuring before committing.
WHY BENCHMARK
To know whether a change helps, rather than believing it does.
WHAT MAKES A BENCHMARK VALID
Realistic data volume Realistic query mix Realistic concurrency A warmed cache, or a deliberately cold one
WHY THE CACHE MATTERS
The first run reads from disk and the second from memory, differing enormously.
WHAT TO DECIDE
Which you are measuring.
WHAT MAKES A BENCHMARK WORTHLESS
A tiny dataset One query repeated Measuring on a machine doing other work Comparing runs with different data
WHAT TO MEASURE
Response time at percentiles Throughput Resource use during the test
WHY PERCENTILES AGAIN
The slowest queries are what users notice.
WHAT TO CHANGE BETWEEN RUNS
One thing.
WHAT TO RECORD
The full configuration and data state.
WHY
Results are meaningless without knowing what was tested.
WHAT TO BE SCEPTICAL OF
Published benchmarks comparing databases.
WHY
They are usually configured to favour one, and your workload differs.
WHAT TO BENCHMARK INSTEAD
Your own queries, on your own data.
WHAT TO DO WITH THE RESULT
Keep it, as a baseline to compare against later.
