Blog · Guide
How to make your website agent ready: a 10-step checklist
A practical checklist to make your website readable for ChatGPT, Claude, Perplexity and AI shopping agents: crawler access, server-rendered HTML, llms.txt, structured data and LLM-ready content, with a test for each step.
By Sascha Hoffmann · Published · Markdown
Making your website agent ready means an AI agent can reach your pages, read the important facts without running JavaScript, and find a clear path to act: buy, book or get in touch. Most of the work is a handful of technical fixes your developer can ship in a day.
This checklist goes in the order an agent hits your site. Each step has a test you can run yourself. If you want the whole list checked at once, the free AgentReady Check runs it in under a minute and gives you an agent readiness score.
1. Let AI crawlers in through robots.txt
Open yourdomain.com/robots.txt and look for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, plus any User-agent: * rule that disallows everything.
If you want to opt out of model training, block the training crawlers only and keep the search and user-triggered agents allowed. Blocking all of them removes you from AI answers.
Test: search the file for each crawler name. Details and copy-paste rules: How to allow GPTBot and other AI crawlers in robots.txt.
2. Make sure your firewall agrees with robots.txt
Bot protection, CDN rules and rate limits often block AI crawlers that robots.txt allows. Every AI crawler is datacenter traffic, which many security rules treat as suspicious.
Test: request your homepage with a crawler user agent and compare the status code with a normal browser request:
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.2" https://yourdomain.com/
A 403, 429 or 503 where a browser gets 200 means agents are locked out.
3. Serve your content as HTML, not only JavaScript
Most AI crawlers do not run JavaScript. If your product descriptions, prices or FAQ load after hydration, they do not exist for the agent.
Test: view the page source (not the inspector) and search for a key sentence. If it is missing, switch that page to server-side rendering, static generation or prerendering. Guide: How to make a JavaScript site readable to AI crawlers.
4. Publish a sitemap and canonical URLs
A sitemap.xml that robots.txt points to, plus a rel="canonical" tag on every page, tells agents which pages exist and which URL is the real one.
Test: open yourdomain.com/sitemap.xml, then check that robots.txt contains a Sitemap: line pointing to it.
5. Verify your domain with Google and Bing
AI search products draw on search indexes, and ChatGPT search in particular has used Bing. Verifying your domain in Google Search Console and Bing Webmaster Tools and submitting your sitemap there is cheap insurance.
Test: both consoles should show your sitemap as read without errors.
6. Add an llms.txt file
llms.txt is a short Markdown file at the root of your domain that tells language models what you offer and links to the pages with details and prices. It takes an hour to write and is one of the most commonly missing signals in our scans.
Test: yourdomain.com/llms.txt should return plain text with status 200, not your HTML 404 page. Guide: What is llms.txt, and is it worth adding?
7. Add structured data with real numbers
JSON-LD gives agents explicit fields instead of making them parse your layout. Start with Organization on the homepage, Product and Offer on product and pricing pages, and FAQPage where you answer questions. Store prices as numbers with a currency, not as "from $49".
Test: run a page through the Schema Markup Validator or Google's Rich Results Test. Product fields that matter most to agents: How AI agents choose between products.
8. Write LLM-ready content
LLM-ready content is text a model can lift and quote without losing meaning. In practice:
- Put the answer in the first two sentences under each heading.
- Use headings phrased as the questions buyers ask.
- State prices, limits, delivery areas and differences to competitors in plain sentences.
- Keep one topic per page and avoid burying facts in images, tabs or accordions that need JavaScript.
- Keep the HTML lean. Heavy markup eats the space a model has for your actual text: How HTML size affects what an AI model can read.
Test: paste a section into ChatGPT and ask it to state your price and main difference to a competitor. If it hedges, the text is not explicit enough.
9. Give agents a way to act
Agents increasingly complete tasks, not just answer questions. Make the path short and machine-readable: a public pricing page, a checkout that works without pop-ups and hidden steps, and, if you have one, a documented API or MCP server. The protocols are still moving; where they stand is in Agentic checkout in 2026.
Test: ask a browser agent to find your price and start a purchase or booking. Watch where it stops.
10. Re-check after every change
Relaunches, CMS updates and new bot-protection rules break agent access quietly. Nobody gets an error; agents just answer with less information about you, as we described in When an AI agent cannot read your page.
Test: re-run the free check after each release, or call it from your CI through the API.
The short version
| Step | What to check | Where |
|---|---|---|
| 1 | AI crawlers allowed | robots.txt |
| 2 | Same status code for crawlers and browsers | CDN, firewall |
| 3 | Content in the raw HTML | page source |
| 4 | Sitemap and canonical URLs | sitemap.xml, head |
| 5 | Google and Bing verification | webmaster consoles |
| 6 | llms.txt | domain root |
| 7 | JSON-LD with numeric prices | page head |
| 8 | Answer-first, explicit copy | content |
| 9 | Short path to buy or book | checkout, API |
| 10 | Re-test after releases | free check, API |
Technical readiness is the floor, not the finish. Once agents can read you, the open question is whether they recommend you when a buyer asks. AgentReady Monitor tracks that each month across the major models.
FAQ
How long does it take to make a website agent ready?
For most sites, steps 1 to 7 are a day or two of developer work. Content changes in step 8 take longer and pay off page by page.
Do I need an llms.txt file if I already have a sitemap?
They do different jobs. A sitemap lists URLs; llms.txt explains what you offer and which pages matter. Adding one is cheap.
Should I block GPTBot?
Only if you want to opt out of OpenAI model training. Keep OAI-SearchBot and user-triggered agents allowed, or you risk not appearing in ChatGPT search answers.
Is there a free tool to check if my site is agent ready?
Yes. The AgentReady Check scans nine areas and returns an agent readiness score from 0 to 100, free and without a login.