When an AI agent cannot read your page, it still gives an answer about you

When an AI agent fails to fetch your page, it usually does not report an error. It evaluates whatever it received, which may be nothing, and moves on with a perfectly well-formed verdict.

I learned this from the builder's side of the table, and it changed how I think about bot protection on the sites I run.

The short answer

Check what your site returns to agent-like requests, not just which status code. A 403, a bot challenge served with a 200, a consent wall or an empty JavaScript shell all look the same to an agent: a page with no answer on it.

Then decide deliberately which of those you want. Most sites never made that decision. Their security layer made it for them.

What happened when my own agent got blocked

In September I built a small triage tool that scores search queries against the pages that rank for them. It fetches each page, writes a short summary and asks a decision model, TypeSafe's Jev, how well the page answers the query on a scale from 0 to 3.

On the first real run, every one of the 26 page fetches failed. My tool sent a polite user agent that included its own name, and the site's host answered each request with a 403. It made no difference whether the requests ran in parallel or one after the other. The name was the problem.

So every page summary was empty. Jev did exactly what it should with an empty page: it scored the fit at roughly zero for all 117 rows. The question "does this topic deserve its own page?" then fired on 106 of them.

There was no exception, no warning and no failed call. The table filled up, the progress bar finished, and the result looked like a finding. Only the absurdity of 106 new pages out of 117 rows gave it away.

After the fix it was 4.

Why modern agents fail silently

That behaviour is not a bug in one model. It follows from how agents are being built right now.

Models designed for decisions guarantee a valid answer shape. Jev, for example, can only return values from the options you define, which is exactly why it is useful inside software. TypeSafe describes the flip side itself: the guarantee is about the schema, not the truth. Feed it an empty page and you get a valid, confident answer about an empty page.

Language models behave similarly in agent pipelines. Asked to summarise a challenge page, they summarise the challenge page. Asked whether that summary matches a buyer's question, the answer is no.

From your side, the outcome is the same in every case. Somewhere in an agent's run, your brand was evaluated, found irrelevant, and dropped from a shortlist. Nothing in your logs says why, and the agent's user never finds out that the comparison was incomplete.

The five ways a page comes back empty

A blocked request is only the most obvious case. These are the patterns I would test for, roughly in order of how often I see them.

1. A 403 from a firewall or host. Rules that match on user agent strings, datacenter IP ranges or request rates. Your robots.txt has no say here. Our crawler guide covers the named AI crawlers; this is about everything else.
2. A challenge page served with 200. This is worse than a 403, because the status looks healthy. The HTML is a "checking your browser" interstitial, and a naive agent reads it as your content.
3. A consent or cookie wall that replaces the content. A banner that overlays the page is fine. A banner that is the page until someone clicks is, for an agent, a page about cookies.
4. Content that only exists after JavaScript runs. The HTML is a shell with a loading spinner. The JavaScript rendering guide covers the details.
5. Login or region redirects. Prices behind an account, or an automatic redirect to a country page that does not carry the product.

Why honest agents get blocked first

Here is the uncomfortable part. The agents most likely to hit your firewall are often the most transparent ones.

Large platforms send well-known, verifiable user agents, and many security products have allowlists for them. Agentic browsers mostly look like regular browsers. The requests that stand out are custom agents built by developers and companies, which, like mine, announce what they are in the user agent string.

That is exactly the population building the next generation of buying and research agents. Blocking by name is blocking by honesty.

How to test what agents receive

A status code alone is not enough, because of the challenge-page case. Compare status, size and title across a few user agents:

#!/usr/bin/env bash
# Compare what different clients receive from the same URL.
URL="https://example.com/products/some-product"
UAS=(
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36"
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
  "Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com)"
  "my-research-agent/0.1 (+https://example.org/agent)"
)
for UA in "${UAS[@]}"; do
  BODY=$(curl -s -L -A "$UA" -w '\n%{http_code}' "$URL")
  CODE=$(tail -n1 <<<"$BODY")
  SIZE=$(head -n -1 <<<"$BODY" | wc -c)
  TITLE=$(head -n -1 <<<"$BODY" | grep -o '<title>[^<]*' | head -n1 | sed 's/<title>//')
  printf '%s | %6s bytes | %s | %s\n' "$CODE" "$SIZE" "$TITLE" "${UA:0:40}"
done

The last user agent is the important one. It stands for any honest custom agent. If the browser row returns 80 KB and your product title while that row returns 4 KB and "Just a moment", you have found your silent failure.

The user agent strings above are illustrations. Operators change them, so check the current values in each operator's documentation before you build allowlists on them.

What to change

You do not have to choose between security and being readable. You have to make the choice consciously.

• Allowlist verified operators in your firewall where your provider supports verified bot categories, instead of relying on user agent strings alone.
• Rate-limit instead of block for unknown agents on public pages. A product page read ten times a minute is not an attack.
• Keep challenges for actions, not for reading. Put friction on login, checkout and forms, and leave public product and pricing pages fetchable.
• Render content behind the consent banner, not instead of it. Consent is a legal requirement for tracking in Europe, not for showing a price.
• Watch your logs for 403s to non-browser clients on product URLs. A spike there is lost evaluations, not only saved bandwidth.

What I changed in my own tool

On the builder side, the lesson was just as clear. My triage now refuses to score a page whose fetch failed or whose summary is empty, records it as skipped, and stops the run if an implausibly large share of rows gets the same verdict.

I would like every agent to work like that. Many will not, at least not soon. Until then, the only reliable fix is on your side: make sure that what an agent receives from your site is the page you meant to publish.

AgentReady Check runs live requests with named AI user agents against your homepage and shows where a real crawler gets a different answer than a browser.