Skip to content
nsin docs
EN فا

AI crawler traffic, in full

GET
/analytics/ai-crawlers
 
curl --request GET \
  --url 'https://api.nsin.ir/analytics/ai-crawlers?domain=example.com&period=1h&crawler=gptbot' \
  --header 'Authorization: Bearer <token>'

Everything the AI Crawl Control pages are built from, in one response: period counters, a timeseries, a per-crawler table, and the paths crawlers read or were refused.

summary, series, top_paths and blocked_paths cover AI crawlers only — the kinds listed in ai_kinds, or the single kind named by crawler. Classic search and SEO crawlers (googlebot, bingbot, yandexbot, ahrefsbot, semrushbot, mj12bot, generic-bot) are deliberately left out of those, so they cannot drown the AI numbers. crawlers is the exception: it lists every bot kind actually seen on the domain, so nothing is invisible.

Human traffic never reaches any of these numbers.

The Markdown counters in summary describe the Markdown-for-Agents feature: markdown_answered is what the edge actually rewrote to Markdown, and markdown_missed is the rest of what could plausibly have been Markdown (markdown_eligible — responses below 300). Redirects, 404s and images are not counted against the feature.

Authorizations

Section titled “Authorizations ”

Parameters

Section titled “ Parameters ”

Query Parameters

Section titled “Query Parameters ”
domain
required
string

The domain name (for example example.com). These endpoints take the domain as a query parameter rather than a path segment.

Example 
example.com
period
string
default: 24h
Allowed values: 1h 3h 6h 12h 24h 48h 7d 30d

Time window, ending now. Time series are bucketed by minute for 1h, by hour up to 48h, and by day for 7d and 30d. Buckets are cut on Asia/Tehran (UTC+03:30) boundaries and emitted as UTC instants, so an hourly bucket starts at :30 UTC. An unrecognised value falls back to 24h.

crawler
string

Narrow every panel except crawlers to one bot kind, from the canonical catalog — the values in ai_kinds, plus googlebot, bingbot, duckduckbot, yandexbot, ahrefsbot, semrushbot, mj12bot and generic-bot. Omit for all AI crawlers.

Example 
gptbot
hostname
string

Narrow to one subdomain. Matches the exact host, any subdomain of it, or a bare label — so example.com matches api.example.com, and api matches api.example.com, but exam matches neither.

path
string

Narrow to a URL path prefix.

Responses

Section titled “ Responses ”

200

Section titled “200 ”

AI crawler activity for the period.

Media type application/json
object
summary
object
requests
integer
allowed

Requests answered with a status below 400.

integer
unsuccessful

Requests answered with 4xx or 5xx.

integer
bytes

Bytes served to crawlers.

integer
markdown_answered

Responses the edge rewrote to Markdown.

integer
markdown_missed

Eligible responses that were not rewritten.

integer
markdown_eligible

Responses that could plausibly have been Markdown — status below 300.

integer
crawlers

Every bot kind seen, busiest first — not limited to AI crawlers.

Array<object>
object
kind

Bot kind, e.g. gptbot or claudebot.

string
requests
integer
allowed
integer
unsuccessful
integer
bytes
integer
markdown

Responses served to this crawler as Markdown.

integer
last_seen

Most recent request from this crawler in the period.

string format: date-time
series
Array<object>
object
timestamp
string format: date-time
requests
integer
allowed
integer
unsuccessful
integer
bytes
integer
s2xx
integer
s3xx
integer
s4xx
integer
s5xx
integer
top_paths

Top 10 paths AI crawlers read successfully.

Array<object>
object
path

URL path, without the query string.

string
hostname
string
requests
integer
blocked_paths

Top 10 paths AI crawlers asked for and did not get (4xx or 5xx) — the content agents want but cannot cite.

Array<object>
object
path

URL path, without the query string.

string
hostname
string
requests
integer
ai_kinds

The bot kinds counted as AI crawler traffic.

Array<string>
Example generated
{
  "summary": {
    "requests": 1,
    "allowed": 1,
    "unsuccessful": 1,
    "bytes": 1,
    "markdown_answered": 1,
    "markdown_missed": 1,
    "markdown_eligible": 1
  },
  "crawlers": [
    {
      "kind": "example",
      "requests": 1,
      "allowed": 1,
      "unsuccessful": 1,
      "bytes": 1,
      "markdown": 1,
      "last_seen": "2026-04-15T12:00:00Z"
    }
  ],
  "series": [
    {
      "timestamp": "2026-04-15T12:00:00Z",
      "requests": 1,
      "allowed": 1,
      "unsuccessful": 1,
      "bytes": 1,
      "s2xx": 1,
      "s3xx": 1,
      "s4xx": 1,
      "s5xx": 1
    }
  ],
  "top_paths": [
    {
      "path": "example",
      "hostname": "example",
      "requests": 1
    }
  ],
  "blocked_paths": [
    {
      "path": "example",
      "hostname": "example",
      "requests": 1
    }
  ],
  "ai_kinds": [
    "example"
  ]
}

400

Section titled “400 ”

The domain query parameter is missing, or crawler is not a known bot kind.

Media type application/json

The single error shape used by every endpoint.

object
error
required

Human-readable description of what went wrong.

string
Examples
Example unknownCrawler 
{
  "error": "unknown crawler"
}

401

Section titled “401 ”

Missing, malformed, revoked or expired API key. (A valid key whose account has been deactivated gets 403 account is inactive instead.)

Media type application/json

The single error shape used by every endpoint.

object
error
required

Human-readable description of what went wrong.

string
Examples
Example invalidKey 
{
  "error": "invalid API key"
}

402

Section titled “402 ”

The domain has no active plan at all, so no plan feature is available on it. Buy or renew a plan for the domain. (A plan that is active but does not include the feature answers 403 instead.)

Media type application/json

The single error shape used by every endpoint.

object
error
required

Human-readable description of what went wrong.

string
Examples
Example noPlan 
{
  "error": "This domain does not have an active plan."
}

403

Section titled “403 ”

The domain’s plan does not include the feature this endpoint needs (monitoring for most sections, logs for raw and top-N request data).

Media type application/json

The single error shape used by every endpoint.

object
error
required

Human-readable description of what went wrong.

string
Example 
{
  "error": "read-only API key"
}

404

Section titled “404 ”

No such domain, or it is not visible to this account. Domains you cannot access are reported as not found rather than forbidden.

Media type application/json

The single error shape used by every endpoint.

object
error
required

Human-readable description of what went wrong.

string
Example 
{
  "error": "read-only API key"
}

429

Section titled “429 ”

The key exceeded its request budget (300 requests per minute by default).

Media type application/json

The single error shape used by every endpoint.

object
error
required

Human-readable description of what went wrong.

string
Examples
Example limited 
{
  "error": "rate limit exceeded"
}

503

Section titled “503 ”

Either the analytics backend is temporarily unreachable (retry; no data is lost), or the answer is still being computed. The per-domain sections and the cached account-wide views (domains-overview, global-summary, global-summary/peak, requests-compare, bandwidth-overview, nodes-overview) wait at most about 25 seconds for their query; past that they answer 503 with pending: true and a Retry-After header, while the query keeps running and caches its result, so the retry is served from that cache instead of starting the same scan again. Identical concurrent requests share one computation.

Media type application/json
object
error
required

Human-readable description of what went wrong.

string
pending

true when the query is still running rather than failing. Retry after Retry-After seconds.

boolean
Examples
{
  "error": "analytics unavailable"
}

Headers

Section titled “Headers ”
Retry-After
integer
Example 
5

Seconds to wait before retrying. Sent only with pending.

504

Section titled “504 ”

The query ran out of time for this period. Retrying the same request runs the same query again, so ask for a shorter period (or narrow it with hostname/path) instead.

Media type application/json

The single error shape used by every endpoint.

object
error
required

Human-readable description of what went wrong.

string
Examples
Example timedOut 
{
  "error": "summary timed out for this period; try a shorter one"
}