AI crawler access
11/18
AI crawlers get through in principle, but your access rules are not a deliberate choice.
robots.txt
robots.txt found (21 lines). Rules are being evaluated.GPTBot
GPTBot is named explicitly in your robots.txt and allowed on the homepage.ClaudeBot
ClaudeBot is named explicitly in your robots.txt and allowed on the homepage.PerplexityBot
PerplexityBot is named explicitly in your robots.txt and allowed on the homepage.Google-Extended
Google-Extended is not mentioned in your robots.txt — access is ungoverned. It is allowed, but only by default rather than by decision.Live test GPTBot
With the GPTBot user agent your site answers 403, versus 200 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Live test ClaudeBot
With the ClaudeBot user agent your site answers 403, versus 200 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Live test PerplexityBot
With the PerplexityBot user agent your site answers 404, versus 200 for a browser. That is active blocking — typically a CDN rule or a bot firewall.Google-Extended
Google-Extended is only a robots.txt token, not a crawler of its own, so there is no live test for it. It controls whether Gemini may use your content.
Next step
Name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended explicitly in your robots.txt with "Allow: /" — and check your CDN or firewall so no bot rule intercepts them first.