A small file with large consequences.
WHAT IT DOES
Tells crawlers which paths they may request. It controls crawling, not indexing, which is a distinction that matters.
A page blocked in robots.txt can still appear in results if other sites link to it, because the engine knows it exists without being able to read it.
WHERE IT LIVES
At the root: yourdomain.com/robots.txt
WHAT TO BLOCK
Admin areas Internal search result pages, which waste crawling Faceted filter URLs generating infinite combinations Cart and checkout
WHAT NOT TO BLOCK
CSS and JavaScript files, which engines need to render your pages Pages you want indexed Your entire site, which is the classic disaster
THE MOST COMMON MISTAKE
A staging robots.txt disallowing everything, copied to production. The site disappears from search within weeks.
Check yours after any migration.
TO PREVENT INDEXING
Use a noindex directive on the page, not robots.txt. Blocking crawling prevents the engine seeing the noindex.
TESTING
Search Console provides a robots.txt tester.
