Pipelines feeding language models.
WHAT RETRIEVAL-BASED APPLICATIONS REQUIRE
A corpus of documents Splitting into passages Embedding those passages Storing and indexing them Retrieving relevant ones at query time
WHAT CHUNKING IS
Dividing documents into passages of a workable size.
WHY IT MATTERS MOST
Chunks that split a concept in half retrieve poorly, and this is the commonest cause of bad results.
WHAT TO PREFER
Splitting on structure: sections, paragraphs, headings.
WHAT TO RETAIN WITH EVERY CHUNK
Its source document Its position Any structural context, such as headings above it Access permissions
WHY PERMISSIONS SPECIFICALLY
Retrieval must not return content the asker may not see.
WHAT AN EMBEDDING IS
A numerical representation of meaning.
WHAT CHANGING THE EMBEDDING MODEL REQUIRES
Reprocessing the entire corpus.
WHY
Representations from different models are not comparable.
WHAT TO PLAN
That reprocessing, before it is needed.
WHAT TO BUILD
Incremental updating, so changed documents are reprocessed rather than everything.
WHAT TO MONITOR
Whether retrieved passages are actually relevant.
