Storage and movement.
WHAT THE CONSTRAINT FREQUENTLY IS
Data movement, not computation.
WHY
Moving data costs time and energy, increasingly more than computing on it.
WHAT THAT ARGUES FOR
Computing where the data is, rather than moving it.
WHAT STORAGE TIERS EXIST
Node-local, fast and temporary Parallel shared storage, for active work Archive storage, slow and large
WHAT TO USE FOR WHAT
Node-local for temporary files Shared for input and output Archive for retention
WHAT NOT TO DO
Leave results on fast shared storage indefinitely.
WHY
It is expensive and frequently subject to purging policies.
WHAT TO PLAN
How results are moved to archive, and how they are found later.
WHAT METADATA TO RECORD
What produced the data, when, with what parameters and versions.
WHY
Data nobody can attribute is data nobody can use.
WHAT TO BE CAREFUL WITH
Many small files, which perform badly on parallel file systems Reading the same data repeatedly rather than caching it
WHAT TO CHECK BEFORE A LARGE RUN
That the output will fit, and that quota allows it.
