Natural language processing practice.
WHAT TOKENISATION DOES
Splits text into units the model processes.
WHAT SUBWORD TOKENISATION PROVIDES
Handling of unseen words, and a manageable vocabulary.
WHY THAT MATTERS
Whole-word vocabularies cannot handle new words at all.
WHAT AN EMBEDDING IS
A learned numerical representation of a token or text.
WHAT SENTENCE EMBEDDINGS PROVIDE
Comparison of meaning between texts.
WHAT THE COMMON TASKS ARE
Classification Named entity recognition Similarity and search Summarisation Translation Question answering
WHAT TO TRY FIRST FOR CLASSIFICATION
A simple model on term-frequency features.
WHY
It is fast, interpretable, and frequently competitive on narrow tasks.
WHAT TO TRY NEXT
A pre-trained model, fine-tuned.
WHAT TO BE CAREFUL WITH
Text that is not in the language the model was trained on Code-switching between languages Domain vocabulary Informal spelling and abbreviation
WHY THAT MATTERS HERE
Local English varieties, Nigerian languages and code-switching are all poorly represented in most models.
WHAT TO DO
Evaluate on your own text, and fine-tune where performance is inadequate.
WHAT TO NEVER ASSUME
That benchmark performance applies to your users' language.
