How they work.
WHAT A NEURON COMPUTES
A weighted sum of inputs, plus a bias, passed through a non-linear function.
WHY THE NON-LINEARITY MATTERS
Without it, any depth of network collapses to a single linear transformation.
WHAT A LAYER IS
A set of neurons applied in parallel.
WHAT DEPTH PROVIDES
Composition: later layers combining features learned by earlier ones.
WHAT BACKPROPAGATION DOES
Computes how much each parameter contributed to the error.
WHAT GRADIENT DESCENT DOES
Adjusts parameters in the direction reducing error.
WHAT THE COMMON ACTIVATION FUNCTIONS ARE
A rectifier, zero below zero and linear above Variants avoiding its dead-neuron problem Smooth functions used in newer architectures
WHAT VANISHING GRADIENTS ARE
Gradients becoming so small that early layers stop learning.
WHAT ADDRESSES IT
Better activations Normalisation between layers Connections skipping layers
WHAT SKIP CONNECTIONS PROVIDE
A path for gradients to reach early layers directly.
WHY THAT MATTERED HISTORICALLY
It made very deep networks trainable at all.
WHAT NORMALISATION DOES
Stabilises the distribution of values between layers.
WHAT TO UNDERSTAND BEFORE BUILDING
That most work is choosing an existing architecture, not inventing one.
