AI Crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended)

AI crawlers are the automated bots AI companies use to read websites, for two distinct purposes: collecting training data for future models, and fetching current pages to ground live answers. Each company names its bots and publishes how to control them through robots.txt. OpenAI runs GPTBot (training) and OAI-SearchBot (live search for ChatGPT); Anthropic runs ClaudeBot; Perplexity runs PerplexityBot; Google uses Google-Extended as a control for AI training while its ordinary Googlebot continues to serve search, AI Overviews and AI Mode.

The problem for stores is that blocking these bots has become a default in many places — hosting platforms, content-delivery networks like Cloudflare, security apps and "block AI" toggles often shut them all out at once, sometimes without the store owner knowing. Declining to feed training data is a reasonable choice. Blocking the search-time crawlers is a different decision: it means opting out of being recommended by that tool, because a bot that can't read your page can't cite it. A crawl check that finds AI crawlers blocked is reported as an on-site defect for that reason. The fix is deliberate: allow the search crawlers you want recommending you, and decide about training crawlers separately.