Text, documents, images and audio.
WHAT MAKES IT DIFFERENT
No schema Large individual objects Meaning requiring extraction
WHERE TO STORE IT
Object storage, with metadata in a database.
WHY NOT IN THE DATABASE
Large objects in a database degrade performance and complicate backup.
WHAT METADATA TO RECORD
Location, type, size, source, when acquired, and any extracted attributes.
WHAT EXTRACTION PRODUCES
Text from documents Transcripts from audio Labels and detections from images Structured fields from forms
WHAT TO STORE
Both the original and the extracted result.
WHY
Extraction methods improve, and reprocessing requires the original.
WHAT TO VERSION
The extraction method, alongside its output.
WHY
So results can be attributed and compared.
WHAT TO BE CAREFUL WITH
File formats varying more than expected Encoding problems in extracted text Very large files exhausting memory Personal data inside documents
WHAT THAT LAST POINT REQUIRES
Treating extracted text with the same care as structured personal data.
WHAT TO PROCESS
Incrementally, since reprocessing everything is expensive.
