The summary.
MOST WORK IS ADAPTING AN EXISTING ARCHITECTURE, NOT INVENTING ONE
Start from a pre-trained model, always. It turns a problem needing millions of examples into one needing hundreds.
Match the original preprocessing exactly — different normalisation degrades performance silently.
AUGMENTATION IS THE MOST EFFECTIVE REGULARISER FOR IMAGE TASKS
Provided the augmentations reflect real variation and do not change the label.
ATTENTION REMOVED BOTH LIMITS OF RECURRENT MODELS
Distance stopped mattering and computation parallelised, which is why transformers spread from language into vision, audio, time series and biology.
Its cost scales with the square of sequence length, which is what limits context.
BENCHMARK PERFORMANCE DOES NOT APPLY TO YOUR USERS' LANGUAGE
Local English varieties, Nigerian languages and code-switching are poorly represented. Evaluate on your own text.
TRY PROMPTING BEFORE FINE-TUNING, AND RETRIEVAL BEFORE BOTH FOR KNOWLEDGE
Knowledge changes; retrieval updates instantly and fine-tuning does not.
BUILD THE EVALUATION SET BEFORE THE SYSTEM
Without it, every change is judged by impression. For retrieval systems, faithfulness to sources is the metric that matters most.
