Reducing load from requests you do not want.
IDENTIFYING IT
The access log. Extracting and counting addresses shows which make disproportionate requests. Counting user agents shows which tools.
Compare server-reported traffic with analytics. A large gap means most requests are automated.
BLOCKING BY ADDRESS
cPanel IP Blocker, or a deny rule in .htaccess.
Effective against a single persistent source, useless against distributed traffic from thousands of addresses.
BLOCKING BY USER AGENT
An .htaccess rule matching the agent string. Easily evaded, since agents are self-reported, but it stops the honest ones.
BLOCKING EXPENSIVE ENDPOINTS
Search pages, faceted filters and export endpoints are expensive and frequently hammered by crawlers. Disallow them in robots.txt, and rate-limit or block at the server if crawlers ignore it.
XMLRPC
If you run WordPress and do not use remote publishing, block xmlrpc.php. It is heavily abused for brute-force attempts.
THE BETTER ANSWER
A CDN with bot management handles this far more effectively than anything at the account level.
DO NOT BLOCK
Legitimate search engine crawlers. Verify before blocking, since bad bots impersonate them.
