Markdown for Agents
An AI agent asking for one of your pages gets the same HTML a browser does — navigation, scripts, styles, tracking, the lot. Most of that is noise it has to pay for in tokens before it reaches your actual content.
Markdown for Agents converts eligible HTML pages to Markdown at the edge when the client asks for it. Typical result: 5–10× fewer tokens, so much more of your page fits in what the agent actually reads.
Human visitors are completely unaffected — they never negotiate Markdown, so they always get your HTML.
Turning it on
Section titled “Turning it on”- Open your domain in the dashboard.
- Go to AI Crawl Control → Optimization.
- Switch Markdown for Agents on.
It is off by default and needs an active plan.
How it works
Section titled “How it works”It is standard HTTP content negotiation. A client that prefers Markdown says so:
curl -H 'Accept: text/markdown' https://example.ir/pricingIf the page is eligible, nsin converts it and answers with:
| Header | Value |
|---|---|
Content-Type | text/markdown; charset=utf-8 |
X-Markdown-Tokens | tokens in the Markdown it sent |
X-Original-Tokens | tokens the HTML would have cost |
Vary | Accept |
The Vary: Accept matters: it tells every cache along the way that this URL has
two representations, so a browser is never handed the Markdown and an agent is
never handed the HTML.
Conversion happens on both cache hits and live responses, and a hot page is converted once and reused.
When it doesn’t convert
Section titled “When it doesn’t convert”Conversion never turns a working page into an error. If it can’t run, the original HTML is served instead. That happens when:
- the response isn’t
200 text/html— an image, a JSON API, a redirect; - the page is too large to convert within the edge’s budget;
- the HTML can’t be parsed in time, or its encoding isn’t supported.
The request still appears in your logs, with the reason recorded, so you can see which pages fall back.
Checking it works
Section titled “Checking it works”The Optimization page has a tester. Give it a subdomain and a path and it fetches that URL twice — once as a browser, once as an agent — and shows both results side by side:
- what each one got back (status, content type, cache status),
- whether the agent’s copy was converted,
- the token count of each, and the percentage saved.
Below the tester is the live view: which requests negotiated Markdown, which were converted, and which fell back to HTML.
What’s next
Section titled “What’s next”- AI Crawl Control — decide which crawlers reach you in the first place.
- Proxy Headers — the rest of the headers nsin adds.