# Every byte of your HTML competes for the same context window

> Cloudflare strips pages down to Markdown before handing them to AI systems, so more of the content fits in a model's context. What that implies for how you publish.

Guide · Published 2026-08-28 · By Sascha Hoffmann

Buried in Matthew Prince's conversation with Matt Turck on the Mad Podcast, in among the traffic numbers, is a small operational detail that says more about how to publish for AI systems than most articles on the subject.

Describing what Cloudflare does for customers who *want* AI systems to consume their content, Prince explains that they take HTML — with all the cruft around it — and convert it to Markdown, which is far more streamlined. The stated reason is not elegance. It is that you can get more of the actual information into these systems without blowing up their context windows.

That is a resource constraint, and it is not one anybody optimised for.

## The constraint nobody was optimising for

Every model reading your page is working inside a fixed budget. Whatever is spent on your markup is not spent on your meaning.

Think about what a typical page actually ships. A navigation menu repeated on every URL. A cookie banner. A footer with forty links. Inline SVG icons that expand to hundreds of characters each and say nothing. Utility class strings longer than the sentences they wrap. Framework hydration payloads. Analytics snippets. Three variants of every image in a `srcset`.

For a human that is invisible scaffolding — the browser resolves it and you see a page. For a system reading the raw document it is all content, and it all costs the same as your actual argument.

The consequence is unglamorous and worth stating plainly: if the sentence that answers your buyer's question got pushed out or truncated, it cannot be quoted. Not ranked lower. Not present.

I want to be careful here, because this is where the genre usually goes wrong. Nobody publishes a formula weighting markup ratio against citation likelihood, and I am not going to invent one. The mechanism does not need a formula. Content that made it into the window can be used; content that did not, cannot.

## Why the fix is a twin, not a rewrite

The obvious reaction is to go and strip your HTML. Mostly, don't. Your pages carry navigation and analytics because humans need them, and a page optimised purely for token efficiency is a worse page for the majority of your revenue.

The better move is to publish the same content twice, in two formats, and be explicit about which is which:

- `/pricing` — the HTML page, built for people.
- `/pricing.md` — the same content as Markdown, built for machines.

No navigation, no banner, no icons. Headings, paragraphs, tables, links. Typically a fraction of the bytes for the same meaning.

This is not a trick, and it matters that it isn't. Serving different *content* to bots than to humans is cloaking, and it will eventually be treated as such. Serving the same content in a format better suited to the reader is what `Accept` headers were invented for. The distinction is whether a person comparing the two would say they say the same thing.

We do this on this site, and you can check it while you read: [`/index.md`](/index.md), [`/faq.md`](/faq.md), [`/pricing.md`](/pricing.md), and this article at [`/blog/html-cruft-and-context-windows.md`](/blog/html-cruft-and-context-windows.md). Every one is generated from the same source as its HTML page, so they cannot drift apart. That last property is the one that decides whether this works long-term.

## The four rules that keep it honest

**Generate both from one source.** If the Markdown is maintained by hand, it is wrong within a month, and a stale twin is worse than no twin — you have published a contradiction about yourself, which is precisely the failure mode you were trying to avoid.

**Declare the relationship.** Each page should point at its own twin, and only its own:

```html
<link rel="alternate" type="text/markdown" href="/pricing.md">
```

The common mistake is putting one of these in a shared layout, at which point every page on the site advertises the homepage's Markdown as its own. We shipped that bug ourselves and only caught it while building this blog.

**Send the right content type.** `text/markdown; charset=utf-8`. A Markdown file served as `text/html` invites the client to parse it as markup, and a client that has to guess sometimes guesses wrong.

**Keep one canonical URL per thing.** The HTML page is the canonical entity; the `.md` file is the same entity in another format. Put the HTML URL in your sitemap and leave the twin out. Two URLs in a sitemap for one article is a request to be treated as two things, which splits whatever authority the piece earns.

## Where llms.txt fits, and where it does not

These are complementary, and they get conflated constantly.

`llms.txt` is one file at your root describing the *whole site* — what you sell, who for, what it costs, where the important pages are. It is a map. It is small, and it is the single cheapest thing on this list to add.

Markdown twins are the *territory*: the full content of each page, in a format that survives the trip into a context window.

A map without territory sends a model to pages it then struggles to read. Territory without a map means the model has to discover which of your four hundred URLs matter. Publish both. They take an afternoon between them.

## How to verify

Three checks, none of which needs a tool:

```bash
# 1. The twin exists and identifies itself correctly.
curl -sI https://example.com/pricing.md | grep -i content-type
# expect: content-type: text/markdown; charset=utf-8

# 2. Compare the payloads for the same content.
curl -s https://example.com/pricing    | wc -c
curl -s https://example.com/pricing.md | wc -c

# 3. Check the page declares its own twin, not the homepage's.
curl -s https://example.com/pricing | grep 'rel="alternate"'
```

The second one is the number that makes the argument. On most sites the ratio is somewhere between five and twenty to one, and people are consistently surprised by their own result.

The third one is the check nobody runs, which is why that bug is so common.

## The short version

Attention used to be the scarce resource, and design competed for it. For the readers that are now the majority of your traffic, the scarce resource is context, and your markup competes for it against your own argument.

Publishing a clean twin is an afternoon of work, it changes nothing for your human visitors, and it removes an entire class of reason for your best sentence not to make it into the answer.

---

Check whether your own site has this problem: https://check.ai-agent-ready.com/
Measure what AI models answer about you: https://ai-agent-ready.com/pricing

A product by The Autopilot — https://the-autopilot.com