AI crawler access
6/18
AI crawlers are locked out or ignored. That is the most expensive mistake on this list.
robots.txt
Your robots.txt is not cleanly retrievable (status 403). AI crawler access is therefore ungoverned.GPTBot
GPTBot is not mentioned in your robots.txt — access is ungoverned.ClaudeBot
ClaudeBot is not mentioned in your robots.txt — access is ungoverned.PerplexityBot
PerplexityBot is not mentioned in your robots.txt — access is ungoverned.Google-Extended
Google-Extended is not mentioned in your robots.txt — access is ungoverned.Live test GPTBot
With the GPTBot user agent your site answers 403, versus 403 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Live test ClaudeBot
With the ClaudeBot user agent your site answers 403, versus 403 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Live test PerplexityBot
With the PerplexityBot user agent your site answers 403, versus 403 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Google-Extended
Google-Extended is only a robots.txt token, not a crawler of its own, so there is no live test for it. It controls whether Gemini may use your content.
Next step
Name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explicitly in your robots.txt with "Allow: /" — and check your CDN or firewall so no bot rule intercepts them first.