Controlling what is crawled.
WHAT IT IS
A file at the root of your site telling crawlers which areas to avoid.
WHAT IT DOES
Requests that crawlers stay away from certain paths.
Respected by legitimate crawlers, ignored by malicious ones.
WHAT IT DOES NOT DO
Prevent indexing. A blocked page can still appear in results if linked from elsewhere.
To prevent indexing, use a noindex instruction instead.
That distinction causes confusion.
WHAT TO BLOCK
Admin areas Search result pages Filter and parameter pages producing endless variations Anything with no value to searchers
WHAT NEVER TO BLOCK
Your CSS and JavaScript, which search engines need to render the page Pages you want indexed Your whole site, accidentally
THAT LAST ONE
A single misplaced line blocks everything.
Check after any change.
WHERE IT LIVES
Your domain followed by /robots.txt
Load it and read it. Many site owners have never seen theirs.
