# How to make your website agent ready: a 10-step checklist

> A practical checklist to make your website readable for ChatGPT, Claude, Perplexity and AI shopping agents: crawler access, server-rendered HTML, llms.txt, structured data and LLM-ready content, with a test for each step.

Guide · Published 2026-10-19 · By Sascha Hoffmann

Making your website **agent ready** means an AI agent can reach your pages, read the important facts without running JavaScript, and find a clear path to act: buy, book or get in touch. Most of the work is a handful of technical fixes your developer can ship in a day.

This checklist goes in the order an agent hits your site. Each step has a test you can run yourself. If you want the whole list checked at once, the [free AgentReady Check](/check) runs it in under a minute and gives you an [agent readiness score](/blog/what-is-an-agent-readiness-score).

## 1. Let AI crawlers in through robots.txt

Open `yourdomain.com/robots.txt` and look for `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, `PerplexityBot` and `Google-Extended`, plus any `User-agent: *` rule that disallows everything.

If you want to opt out of model training, block the training crawlers only and keep the search and user-triggered agents allowed. Blocking all of them removes you from AI answers.

**Test:** search the file for each crawler name. Details and copy-paste rules: [How to allow GPTBot and other AI crawlers in robots.txt](/blog/fix-ai-crawlers-blocked).

## 2. Make sure your firewall agrees with robots.txt

Bot protection, CDN rules and rate limits often block AI crawlers that robots.txt allows. Every AI crawler is datacenter traffic, which many security rules treat as suspicious.

**Test:** request your homepage with a crawler user agent and compare the status code with a normal browser request:

```bash
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.2" https://yourdomain.com/
```

A 403, 429 or 503 where a browser gets 200 means agents are locked out.

## 3. Serve your content as HTML, not only JavaScript

Most AI crawlers do not run JavaScript. If your product descriptions, prices or FAQ load after hydration, they do not exist for the agent.

**Test:** view the page source (not the inspector) and search for a key sentence. If it is missing, switch that page to server-side rendering, static generation or prerendering. Guide: [How to make a JavaScript site readable to AI crawlers](/blog/fix-js-rendering).

## 4. Publish a sitemap and canonical URLs

A `sitemap.xml` that robots.txt points to, plus a `rel="canonical"` tag on every page, tells agents which pages exist and which URL is the real one.

**Test:** open `yourdomain.com/sitemap.xml`, then check that robots.txt contains a `Sitemap:` line pointing to it.

## 5. Verify your domain with Google and Bing

AI search products draw on search indexes, and ChatGPT search in particular has used Bing. Verifying your domain in Google Search Console and Bing Webmaster Tools and submitting your sitemap there is cheap insurance.

**Test:** both consoles should show your sitemap as read without errors.

## 6. Add an llms.txt file

`llms.txt` is a short Markdown file at the root of your domain that tells language models what you offer and links to the pages with details and prices. It takes an hour to write and is one of the most commonly missing signals in our scans.

**Test:** `yourdomain.com/llms.txt` should return plain text with status 200, not your HTML 404 page. Guide: [What is llms.txt, and is it worth adding?](/blog/fix-no-llms-txt)

## 7. Add structured data with real numbers

JSON-LD gives agents explicit fields instead of making them parse your layout. Start with `Organization` on the homepage, `Product` and `Offer` on product and pricing pages, and `FAQPage` where you answer questions. Store prices as numbers with a currency, not as "from $49".

**Test:** run a page through the Schema Markup Validator or Google's Rich Results Test. Product fields that matter most to agents: [How AI agents choose between products](/blog/machine-readable-product-data).

## 8. Write LLM-ready content

LLM-ready content is text a model can lift and quote without losing meaning. In practice:

- Put the answer in the first two sentences under each heading.
- Use headings phrased as the questions buyers ask.
- State prices, limits, delivery areas and differences to competitors in plain sentences.
- Keep one topic per page and avoid burying facts in images, tabs or accordions that need JavaScript.
- Keep the HTML lean. Heavy markup eats the space a model has for your actual text: [How HTML size affects what an AI model can read](/blog/html-cruft-and-context-windows).

**Test:** paste a section into ChatGPT and ask it to state your price and main difference to a competitor. If it hedges, the text is not explicit enough.

## 9. Give agents a way to act

Agents increasingly complete tasks, not just answer questions. Make the path short and machine-readable: a public pricing page, a checkout that works without pop-ups and hidden steps, and, if you have one, a documented API or MCP server. The protocols are still moving; where they stand is in [Agentic checkout in 2026](/blog/agentic-checkout).

**Test:** ask a browser agent to find your price and start a purchase or booking. Watch where it stops.

## 10. Re-check after every change

Relaunches, CMS updates and new bot-protection rules break agent access quietly. Nobody gets an error; agents just answer with less information about you, as we described in [When an AI agent cannot read your page](/blog/when-agents-cannot-read-your-page).

**Test:** re-run the [free check](/check) after each release, or call it from your CI through the [API](/docs/api).

## The short version

| Step | What to check | Where |
| --- | --- | --- |
| 1 | AI crawlers allowed | robots.txt |
| 2 | Same status code for crawlers and browsers | CDN, firewall |
| 3 | Content in the raw HTML | page source |
| 4 | Sitemap and canonical URLs | sitemap.xml, head |
| 5 | Google and Bing verification | webmaster consoles |
| 6 | llms.txt | domain root |
| 7 | JSON-LD with numeric prices | page head |
| 8 | Answer-first, explicit copy | content |
| 9 | Short path to buy or book | checkout, API |
| 10 | Re-test after releases | free check, API |

Technical readiness is the floor, not the finish. Once agents can read you, the open question is whether they recommend you when a buyer asks. [AgentReady Monitor](/) tracks that each month across the major models.

## FAQ

### How long does it take to make a website agent ready?

For most sites, steps 1 to 7 are a day or two of developer work. Content changes in step 8 take longer and pay off page by page.

### Do I need an llms.txt file if I already have a sitemap?

They do different jobs. A sitemap lists URLs; llms.txt explains what you offer and which pages matter. Adding one is cheap.

### Should I block GPTBot?

Only if you want to opt out of OpenAI model training. Keep OAI-SearchBot and user-triggered agents allowed, or you risk not appearing in ChatGPT search answers.

### Is there a free tool to check if my site is agent ready?

Yes. The [AgentReady Check](/check) scans nine areas and returns an agent readiness score from 0 to 100, free and without a login.

---

Check whether your own site has this problem: https://ai-agent-ready.com/check/
Measure what AI models answer about you: https://ai-agent-ready.com/pricing

A product by The Autopilot · https://the-autopilot.com