robots.txt is the file that states which bot may crawl which part of a site. It does not block indexing; it directs crawling. In the AI era it is also where you decide which AI bots can reach the site.
Bots such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended can be explicitly allowed or blocked here. Businesses that want brand visibility should leave them open; those protecting their content close them. Declaring the sitemap address in the file is also standard.
Bots such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended can be explicitly allowed or blocked here. Businesses that want brand visibility should leave them open; those protecting their content close them. Declaring the sitemap address in the file is also standard.
Bots such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended can be explicitly allowed or blocked here. Businesses that want brand visibility should leave them open; those protecting their content close them. Declaring the sitemap address in the file is also standard.
Knowing the term is not enough; you need to know which lever to pull. In a free 30-minute call we look at your store's actual numbers.
Get a free analysis