Pulling data from services.
WHAT TO ESTABLISH FIRST
Rate limits Pagination approach Whether incremental extraction is supported Whether historical data is available What happens to deleted records
WHY RATE LIMITS MATTER MOST
They determine how long extraction takes, and whether it is feasible at all.
WHAT TO IMPLEMENT
Respect for rate limit responses Retry with increasing delay Resumption from where extraction stopped
WHAT PAGINATION APPROACHES EXIST
Offset-based, which is simple and breaks when data changes mid-extraction Cursor-based, which is stable Time-window based
WHAT TO PREFER
Cursor-based, where offered.
WHAT TO STORE
Raw responses, before parsing.
WHY
So parsing can be corrected without re-extracting, which may be impossible.
WHAT TO HANDLE
Schema changes in responses Fields appearing and disappearing Values changing type
WHAT TO NEVER DO
Assume the response structure is stable.
WHAT TO MONITOR
Extraction duration, which grows Error rates Whether record counts are plausible
WHAT TO PLAN
Credential rotation, and what happens when access is revoked.
