When crawling becomes a constraint.
WHAT IT MEANS
Search engines allocate limited crawling to each site.
WHEN IT MATTERS
Large sites, with many thousands of pages.
WHEN IT DOES NOT
Small sites. A site with fifty pages is crawled fully.
Most small businesses need not think about this.
WHAT WASTES CRAWLING
Filter and parameter combinations Duplicate pages Redirect chains Pages returning errors Very slow responses
WHAT TO DO ON A LARGE SITE
Block pointless parameter pages Fix redirect chains Remove or consolidate duplicates Improve server response time
HOW TO SEE IT
Search Console reports crawl statistics.
WHAT TO LOOK FOR
Crawling spent on pages that do not matter.
FOR A SHOP WITH MANY FILTERS
The most common cause of wasted crawling.
WHAT NOT TO DO
Worry about this on a small site.
Spend the effort on content and speed instead.
