Protecting a server from traffic.
WHAT TO ESTABLISH FIRST
Whether the traffic is malicious, automated but legitimate, or genuine growth.
WHY THAT ORDER
Blocking a search engine or a customer is worse than the load.
WHAT TO LOOK AT
Sources, requested paths and declared clients.
WHAT LEGITIMATE CRAWLING LOOKS LIKE
Many distinct paths, a declared name, respect for delays.
WHAT TO DO ABOUT IT
Crawl delays in the robots file Caching, so requests cost nothing
WHY CACHING IS THE REAL ANSWER
It removes the load entirely rather than asking politely.
WHAT TO DO ABOUT MALICIOUS TRAFFIC
Block at the edge, before it reaches the server.
WHY AT THE EDGE
Blocking on the server still consumes its resources.
WHAT AN EDGE PROVIDER OFFERS
Rate limiting per address Challenges for suspicious requests Caching of everything static
WHAT TO RATE LIMIT SPECIFICALLY
Login endpoints Search functions Anything expensive to generate
WHY SEARCH SPECIFICALLY
It is frequently the most database-intensive operation available to anonymous visitors.
WHAT TO MONITOR AFTER APPLYING LIMITS
Whether legitimate users are affected.
WHAT TO AVOID
Permanent blocks applied automatically with no review Blocking entire countries without reason
WHAT TO DOCUMENT
What is blocked and why.
