The summary.
GRAPHICS PROCESSORS OPTIMISE FOR THROUGHPUT, NOT LATENCY
Which is why branching code performs badly — divergent threads execute both paths and discard half the results.
Memory access patterns determine performance more than arithmetic, and scattered access loses most of the achievable bandwidth.
TRANSFER COST FREQUENTLY EXCEEDS THE BENEFIT
Keep data resident on the device and perform many operations on it. Always measure end-to-end time including transfers.
USE THE VENDOR LIBRARIES BEFORE WRITING KERNELS
They are heavily optimised and hand-written code rarely matches them.
MEASURE FRAME TIME, NOT FRAME RATE
Frame time is linear and reveals variance. Occasional long frames are perceived as stutter, which is worse than a uniformly lower rate.
Establish whether you are processor-limited or graphics-limited before optimising — they need opposite remedies.
FOR MACHINE LEARNING, LOW DEVICE UTILISATION USUALLY MEANS THE DATA PIPELINE IS THE PROBLEM
Fix data loading before touching the model.
ESTABLISH WHETHER YOU WILL GENUINELY TARGET SEVERAL VENDORS
Portability has real cost and is frequently never used.
