AI Crawl Control
AI Crawl Control names every AI crawler visiting your site and lets you set its policy. Find it under AI Crawl Control on your domain.
Who nsin recognises
Section titled “Who nsin recognises”| Category | What it means | Examples |
|---|---|---|
| AI Crawler | Bulk scrape, usually for training. | GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Meta-ExternalAgent, Amazonbot |
| AI Search | Powers an answer engine, cites you. | OAI-SearchBot, PerplexityBot, Applebot |
| AI Assistant | Fetches a page live, on request. | ChatGPT-User, Claude-User, Perplexity-User |
| Search Engine | Classic web search. | Googlebot, Bingbot, DuckDuckBot, YandexBot |
| SEO Tool | Backlink and audit crawlers. | AhrefsBot, SemrushBot, MJ12Bot |
Blocking a trainer costs no traffic. Blocking an answer engine costs its results.
The pages
Section titled “The pages”- Overview — AI traffic by company: 24h / 7d / 30d.
- Metrics — per crawler: requests, allowed, cost.
- Security — each crawler’s policy, plus templates.
- Optimization — Markdown for Agents.
Ready-made policies
Section titled “Ready-made policies”The Security page offers four templates. Applying one creates an ordinary bot rule you can edit in Rules.
| Template | What it does |
|---|---|
| Block AI training crawlers | Blocks scraping bots. Search and answer engines untouched. |
| Allow AI search, block trainers | Keeps crawlers that cite you; blocks scrapers. |
| Block all AI crawlers | Blocks every AI kind. Classic search unaffected. |
| Monitor only (dry run) | Blocks nothing; logs each match as if blocked. |
Setting your own policy
Section titled “Setting your own policy”Go to Rules → Add Rule → Bots, or edit a template’s rule:
- Block — refuse with a status you choose (403 by default).
- Serve different content — e.g. a licensing notice.
- Send to a different server — proxy it to another origin.
- Tag only — let it through, marked in analytics.
Rules can target the whole zone or specific records, and run in dry run first. Full reference: Bots & Challenges.
Verified identity
Section titled “Verified identity”Anyone can put GPTBot in a user-agent string. Verified only matches
crawlers nsin confirmed by forward-confirmed reverse DNS. Use it when
allowing a crawler.
What this does not do
Section titled “What this does not do”- It doesn’t edit
robots.txt— it acts at the edge regardless. - It doesn’t affect human visitors, Googlebot, or Bingbot unless a rule names them.
What’s next
Section titled “What’s next”- Markdown for Agents — low-token pages.
- Bots & Challenges — the rule type.
- Monitoring & Analytics — traffic data.