Non-human visitors.
WHAT VISITS YOUR SITE
Search engine crawlers Monitoring services Scrapers Vulnerability scanners Automated attack tools
HOW MUCH
On many small sites, more than human traffic.
WHAT IT CONSUMES
Real bandwidth, processing and connections.
WHAT TO ALLOW
Legitimate search crawlers Services you use
WHAT TO LIMIT OR BLOCK
Aggressive scrapers Scanners probing for vulnerabilities Anything consuming disproportionate resources
HOW TO IDENTIFY THEM
Server logs show what is requesting what, and how often.
WHAT A SCANNER LOOKS LIKE
Rapid requests for paths that do not exist, matching known software.
WHAT TO DO
Rate limiting Blocking by pattern or source A filtering service in front of the site
WHAT NOT TO DO
Block legitimate crawlers, which removes you from search
WHAT TO CHECK
Whether traffic growth is human before celebrating or upgrading.
