Retrieval and its limits.
WHEN THIS IS APPROPRIATE
An API is offered: use it instead
Data is published for reuse You have permission Your own sites
WHAT TO CHECK FIRST
Whether an API exists The site's terms Its stated crawling preferences
WHAT TO RESPECT
Any stated restrictions A reasonable request rate The load you place on someone else's server
WHY RATE MATTERS
Rapid automated requests are indistinguishable from an attack and will be blocked.
WHAT TO SET
A delay between requests A timeout An identifying user agent with contact details
WHAT TO HANDLE
Pages changing structure Requests being blocked Errors and missing pages
THE STRUCTURE POINT
Anything depending on a page's layout breaks when the layout changes.
Expect that, and fail clearly.
WHAT TO CACHE
Responses, so you do not fetch the same thing repeatedly.
WHAT NOT TO DO
Collect personal data without a basis for doing so Ignore terms or restrictions Place substantial load on a site you do not own
WHAT TO PREFER ALWAYS
An official API.
