When there is no interface at all.
WHAT IT IS
Extracting data from pages intended for people.
WHEN IT IS CONSIDERED
No interface exists, and the data is genuinely needed.
WHAT TO ESTABLISH FIRST
Whether it is permitted.
WHAT TO CHECK
The terms of use Any technical instructions the site publishes Whether the data is personal
WHY THE TERMS MATTER
Scraping is frequently prohibited, and doing it anyway carries legal and access risk.
WHAT TO DO INSTEAD, FIRST
Ask whether an interface exists, or could.
WHY
It is frequently available and simply not advertised.
WHAT MAKES SCRAPING FRAGILE
Any change to the page breaks it No versioning or notice Anti-automation measures Content assembled by scripts rather than delivered
WHAT TO BUILD IF YOU PROCEED
Validation that the extracted data is plausible Alerting when extraction fails Rate limiting, to be a considerate visitor Isolation, so failure affects nothing else
WHY PLAUSIBILITY CHECKS MATTER MOST
A layout change frequently produces wrong data rather than no data.
WHAT TO NEVER DO
Request aggressively Ignore instructions the site publishes Collect personal data without a basis
WHAT TO TREAT IT AS
A temporary measure, pending a proper interface.
