Crawling

Crawling is how search engines and AI tools discover the pages on your site. A bot — Googlebot, Bingbot, GPTBot and the rest — starts from pages it already knows, follows the links on them, reads each page it reaches, and follows the links on those. Your sitemap gives it a list to start from; your robots.txt tells it where it may not go. A page the crawler never reaches doesn't exist as far as the engine is concerned, regardless of how good it is.

Stores hide pages from crawlers by accident all the time: a product in no collection and linked from nowhere; a whole section blocked in robots.txt during a redesign and never unblocked; products behind a password or removed by a sold-out setting; a firewall that challenges bots with a puzzle. On large catalogues there's also crawl budget — the finite attention a crawler gives a site — which thousands of near-duplicate filter and sort URLs can exhaust before the crawler reaches real products. Search Console's Pages report shows what was crawled and why some wasn't, and its URL Inspection tool shows exactly what the crawler saw on any single page.