Reducing unwanted requests.
WHAT YOU WANT
Search engine crawlers, at a reasonable rate.
WHAT YOU DO NOT
Scrapers Vulnerability scanners Spam bots Aggressive crawlers with no benefit
WHAT TO DO FIRST
Caching. It makes bot traffic cheap rather than expensive.
ROBOTS.TXT
Requests crawlers stay away from certain areas.
Respected by legitimate crawlers, ignored by malicious ones.
Useful for reducing crawling of pointless pages: filters, search results, parameters.
BLOCKING BY USER AGENT
Rules in .htaccess blocking known bad agents.
Check carefully before applying. Blocking a search engine by mistake is damaging.
RATE LIMITING
Limits requests per address per period.
USING A CDN OR FIREWALL
Filters before requests reach your server.
The most effective option for persistent problems.
WHAT TO MONITOR
Raw logs, periodically. You will see patterns.
WHEN TO ASK US
If the volume is beyond what you can control.
