Turning meaning into numbers.
THE IDEA
Text converted into a list of numbers representing its meaning.
Similar meanings produce similar numbers, even with different words.
WHY THAT IS USEFUL
It allows searching by meaning rather than by exact words.
A search for "how do I reset my password" finds a document titled "changing your login credentials", which keyword search would miss.
WHERE THEY ARE USED
Semantic search Retrieval for question-answering systems Recommendation Clustering and categorisation Detecting duplicate or near-duplicate content
HOW THEY WORK, ROUGHLY
A model trained on large amounts of text learns to place related concepts near each other in a numerical space.
WHAT THIS ENABLES PRACTICALLY
A support system that finds the right article even when the customer uses different words.
A document collection you can question in plain language.
THE LIMITATION
Similarity is not relevance. Two passages can be similar and neither answer the question.
WHY YOU MIGHT ENCOUNTER THE TERM
Any system searching your documents to feed a language model uses them.
