Reading access and error logs.
WHAT AN ACCESS LOG RECORDS
Every request: source, time, what was asked for, the response code, the size, the referrer and the client.
WHAT AN ERROR LOG RECORDS
Problems the server or application encountered.
WHICH TO READ FIRST
The error log, for faults; the access log, for traffic questions.
WHAT THE RESPONSE CODES MEAN BROADLY
- Two hundreds: success
- Three hundreds: redirection
- Four hundreds: the request was at fault
- Five hundreds: the server was at fault
WHAT TO COUNT WHEN INVESTIGATING LOAD
Requests per source address Requests per requested path Requests per client string
WHY THE CLIENT STRING MATTERS
It distinguishes crawlers from visitors immediately.
WHAT A CRAWLER LOOKS LIKE
Sequential paths, no static assets, a declared name.
WHAT AN ATTACK LOOKS LIKE
Repeated requests to login or administrative paths Requests for files that do not exist on your system A narrow set of sources at high rate
WHAT TO DO WITH FINDINGS
Block genuine abuse, and rate-limit or slow crawlers.
WHAT TO BE CAREFUL ABOUT
Blocking a legitimate search engine.
HOW TO VERIFY A CRAWLER
Confirm the address belongs to who it claims.
WHAT TO ARCHIVE
Logs, for long enough to investigate afterwards.
