AI Crawl Control

AI crawlers are already visiting your site. Some send readers back to you; some only take your pages to train a model. AI Crawl Control names each one, shows you what it took, and lets you set a different policy for each.

Find it under AI Crawl Control on your domain in the dashboard.

Who nsin recognises

Every bot the edge identifies is classified by the company that runs it and by what it is for:

Category	What it means	Examples
AI Crawler	Bulk scrape, typically for model training.	GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended, Meta-ExternalAgent, Amazonbot
AI Search	Powers an answer engine’s index, and cites you.	OAI-SearchBot, PerplexityBot, Applebot
AI Assistant	Fetches a page live because a user asked about it.	ChatGPT-User, Claude-User, Perplexity-User
Search Engine	Classic web search.	Googlebot, Bingbot, DuckDuckBot, YandexBot
SEO Tool	Commercial backlink and audit crawlers.	AhrefsBot, SemrushBot, MJ12Bot

The distinction matters. Blocking a training crawler costs you nothing in traffic; blocking an answer engine removes you from the results it shows people.

The pages

• Overview — how much AI traffic reached the domain, grouped by the company behind it, over the last 24 hours, 7 days or 30 days.
• Metrics — the same traffic broken down per crawler: requests, how many were allowed, and what they cost you in bandwidth.
• Security — the table of every crawler with its current policy, and the ready-made templates below.
• Optimization — Markdown for Agents.

Ready-made policies

The Security page offers four starting points. Applying one creates an ordinary bot rule — nothing hidden — which you can then edit or delete in Rules like any other.

Template	What it does
Block AI training crawlers	Blocks the bots that scrape to train models. Search engines and answer engines are untouched.
Allow AI search, block trainers	Keeps the assistants and answer engines that cite you and send readers back; blocks the pure training scrapers.
Block all AI crawlers	Blocks every AI kind, including live assistant fetches. Classic search still indexes you, so Google and Bing rankings are unaffected.
Monitor only (dry run)	Matches every AI crawler and blocks nothing — each match is logged as if it had been blocked.

Tip

Start with Monitor only. A few days of dry-run data tells you exactly which crawlers a policy would have stopped, and how much traffic that is, before anything actually changes.

Setting your own policy

Any rule you can build for a bot, you can build for an AI crawler. Go to Rules → Add Rule → Bots, or edit the rule a template created. The actions are:

• Block — refuse the request with a status you choose (403 by default).
• Serve different content — return your own short response instead of hitting your server. Useful for a licensing notice.
• Send to a different server — proxy the crawler to another origin, so heavy crawling never touches your main one.
• Tag only — let it through, but mark it in your analytics.

Rules can cover the whole zone or only specific proxied records, and can run in dry run before you enforce them. See Bots & Challenges for the full rule reference.

Verified identity

Anyone can put GPTBot in a user-agent string. Switch on verified only and the rule matches only crawlers whose identity nsin has actually confirmed at the edge, by forward-confirmed reverse DNS.

Use it whenever you are allowing a crawler — otherwise you also let in every scraper that copies the name.

What this does not do

• It does not edit your robots.txt. A well-behaved crawler obeys robots.txt; AI Crawl Control acts at the edge, so it applies whether or not the crawler chooses to read it.
• It does not affect human visitors, and it does not affect Googlebot or Bingbot unless you explicitly include them in a rule.

What’s next

• Markdown for Agents — serve agents a clean, low-token version of your pages.
• Bots & Challenges — the underlying rule type.
• Monitoring & Analytics — where the rest of your traffic data lives.