AI crawler access
8/18
AI crawlers get through in principle, but your access rules are not a deliberate choice.
robots.txt
Your robots.txt is not cleanly retrievable (status 429). AI crawler access is therefore ungoverned.GPTBot
GPTBot is not mentioned in your robots.txt — access is ungoverned.ClaudeBot
ClaudeBot is not mentioned in your robots.txt — access is ungoverned.PerplexityBot
PerplexityBot is not mentioned in your robots.txt — access is ungoverned.Google-Extended
Google-Extended is not mentioned in your robots.txt — access is ungoverned.Live test GPTBot
GPTBot gets the same response as a browser (status 200).Live test ClaudeBot
With the ClaudeBot user agent your site answers 429, versus 429 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Live test PerplexityBot
With the PerplexityBot user agent your site answers 429, versus 429 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Google-Extended
Google-Extended is only a robots.txt token, not a crawler of its own, so there is no live test for it. It controls whether Gemini may use your content.
Next step
Name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explicitly in your robots.txt with "Allow: /" — and check your CDN or firewall so no bot rule intercepts them first.