Markdown for Agents
An AI agent gets the same HTML as a browser: nav, scripts, styles, tracking.
Markdown for Agents converts eligible pages to Markdown at the edge on request — typically 5–10× fewer tokens. Human visitors always get HTML.
Turning it on
Section titled “Turning it on”- Open your domain in the dashboard.
- Go to AI Crawl Control → Optimization.
- Switch Markdown for Agents on.
Off by default; needs an active plan.
How it works
Section titled “How it works”Standard HTTP content negotiation:
curl -H 'Accept: text/markdown' https://example.ir/pricingEligible pages get:
| Header | Value |
|---|---|
Content-Type | text/markdown; charset=utf-8 |
X-Markdown-Tokens | tokens in the Markdown it sent |
X-Original-Tokens | tokens the HTML would have cost |
Vary | Accept |
Vary: Accept tells caches this URL has two representations. A hot page
converts once, then reuses.
When it doesn’t convert
Section titled “When it doesn’t convert”Conversion never turns a working page into an error; it falls back when:
- not
200 text/html(image, JSON API, redirect); - too large for the edge’s budget;
- can’t be parsed in time, or unsupported encoding.
It still appears in your logs, with the reason.
Checking it works
Section titled “Checking it works”The Optimization page has a tester: give it a subdomain and path; it fetches the URL as both, showing:
- status, content type, cache status,
- whether the agent’s copy converted,
- token count of each, and percent saved.
Below it, the live view shows which requests negotiated Markdown, converted, or fell back.
What’s next
Section titled “What’s next”- AI Crawl Control — which crawlers reach you.
- Proxy Headers — nsin’s other headers.