Knowledgebase

Making Sure Your Site Can Be Crawled Print

  • seogettingfoundonline, staging, seo, wordpress, security, firewall, password, plugins, performance
  • 0

The prerequisite for everything else.

WHAT CAN PREVENT CRAWLING

robots.txt disallowing the site or important sections A noindex directive, frequently left from a staging environment Password protection Pages reachable only through a form or JavaScript with no crawlable link A slow server, limiting how much is crawled Blocking search engine crawlers at the firewall or in a security plugin

CHECKING

Visit yourdomain.com/robots.txt and read it. A line disallowing everything is the single most common cause of a site being invisible. Use Search Console's URL inspection on an important page to see whether it can be crawled and indexed. View the page source and confirm there is no noindex tag.

THE WORDPRESS SETTING

Settings > Reading has a checkbox discouraging search engines. It is enabled on staging sites and frequently forgotten at launch.

Check it first if a new site is not being indexed.

CRAWL BUDGET

Large sites may not be crawled entirely. Fast pages, a clean sitemap and no wasted crawling on useless URLs all help.


Was this answer helpful?
Back

Are you happy with your experience? Leave us a review on Trustpilot.


Trustpilot