Practical handling.
WHAT THE COMMON DATA TYPES ARE
Raw sequencing reads Aligned reads Variant records Expression matrices Annotations describing genomic features
WHAT THE FILE FORMATS ARE LIKE
Text-based, standardised, and frequently enormous.
WHAT ENORMOUS MEANS
Individual files of many gigabytes; datasets of terabytes.
WHAT THAT REQUIRES
Streaming rather than loading Compression, used throughout Indexed access, so regions can be read without reading everything Adequate storage planning
WHY STREAMING MATTERS MOST
Loading a whole file exhausts memory, and the failure appears only with real data.
WHAT TO BE CAREFUL WITH
Coordinate systems, which differ between formats in whether they start at zero or one Reference genome versions, which must match across every tool Chromosome naming conventions, which differ between sources
WHY THAT SECOND POINT CAUSES SO MUCH TROUBLE
Mixing versions produces results that are wrong but not obviously so.
WHAT TO RECORD
Every reference, version and parameter used.
WHAT TO NEVER MODIFY
Raw data.
WHAT TO PRODUCE INSTEAD
Processed outputs, generated by code.
