robots.txt

robots.txt is a plain-text file at the root of a website — yoursite.com/robots.txt — that tells crawlers which parts of the site they may and may not visit. It's a convention, not a lock: well-behaved bots honour it, and all the ones that matter for visibility do. It controls crawling, not indexing; a blocked page can still appear in results as a bare URL if other sites link to it, and blocking a page is not the way to remove it from search.

Platforms write a sensible default that allows search engines everywhere they should be. The trouble is additions: a developer disallowing /products/ during a redesign and forgetting to remove it; a security app or CDN blocking all bots; and, increasingly, a "block AI" toggle that disallows GPTBot, ClaudeBot, PerplexityBot and Google-Extended in one click — which means opting out of being recommended by those tools. The file takes ten seconds to read in a browser, and a check that finds crawlers blocked is reported as an on-site defect because it undoes every other improvement. Note that firewall and CDN bot rules don't appear in the file and must be checked separately.