Building neural networks.
WHAT THE MAIN OPTIONS ARE
A framework favouring imperative, Python-native definition A framework with a high-level interface and strong deployment tooling Newer functional frameworks favoured in research
WHAT THE FIRST PROVIDES
Models defined as ordinary code, debuggable normally Dominance in research, so new work appears there first
WHAT THE SECOND PROVIDES
A concise high-level interface Mature tooling for serving and mobile deployment
WHAT TO CHOOSE
Whichever your team knows, unless a specific requirement decides it.
WHAT MOST PRACTITIONERS ACTUALLY USE
A higher-level library built on top, reducing repetitive code.
WHAT THOSE PROVIDE
Training loops Mixed precision and distributed training Checkpointing and logging
WHY THAT MATTERS
Hand-written training loops accumulate subtle bugs.
WHAT TO UNDERSTAND REGARDLESS
The computation graph Automatic differentiation Device placement of tensors Where memory is consumed
WHAT CAUSES MOST MEMORY FAILURES
Batch size Retaining tensors that keep the graph alive Accumulating results without detaching them
WHAT TO CHECK FIRST ON A MEMORY ERROR
Whether something is holding the graph.
