Bots & Challenges

Not all automated traffic is bad. Search engines need to crawl you; AI crawlers may or may not be welcome; scrapers and scanners usually aren’t. nsin gives you three tools for this, and they work well together.

Tool	Use it for
Bot rules	Named bots you recognise — search engines, AI crawlers, SEO tools.
Challenges (CAPTCHA)	Traffic you’re unsure about: make it prove it’s a person.
Fingerprint rules	Automated traffic that lies about who it is.

Bot rules

A bot rule matches specific bots that nsin’s edge recognises, and decides what happens to them.

1. Go to Rules → Add Rule → Bots.
2. Pick the bots the rule covers — search-engine crawlers (Googlebot, Bingbot, …), AI crawlers (GPTBot, ClaudeBot, PerplexityBot, …), SEO tools (AhrefsBot, SemrushBot, …), or any bot.
3. Choose the action:
   • Block — refuse the request.
   • Serve different content — return your own response (for example, a short text page) instead of hitting your server.
   • Send to a different server — proxy the bot to another origin, so heavy crawling doesn’t touch your main one.
   • Tag only — let it through, but mark it in your analytics so you can see how much of your traffic it is.
4. Save.

Verified bots only

Anyone can claim to be Googlebot — it’s just a line of text in the request. Switch on verified only and the rule matches only bots whose identity nsin has actually confirmed (by reverse DNS or their published address ranges).

Tip

This is the right way to allow a crawler. Match “verified Googlebot” and you let the real one in without also letting every scraper that copies its name.

Challenges

A challenge rule shows an interactive check to visitors on the paths you choose. Pass it and browsing continues normally; the result is remembered for a while so real people aren’t asked again on every page.

Use a challenge when blocking would be too blunt — a login page under credential-stuffing, a search endpoint being scraped, a signup form collecting spam.

Rate limiting can also end in a challenge rather than a block — see Rate Limiting.

Fingerprint rules

Some automated traffic doesn’t announce itself at all: it sends an ordinary browser user-agent while behaving nothing like a browser. Fingerprint rules match on the shape of the connection itself (its TLS and HTTP fingerprint) rather than on what it claims to be, and can drop it, challenge it, or just tag it.

This is an advanced tool. Start in tag mode, look at what it catches in your analytics, and only then decide whether to challenge or drop.

Order matters

These rules are evaluated in order, so put your allow-style rules (verified search engines) above your broad block-style ones. See Rules Overview.

Common questions

Will blocking AI crawlers hurt my search ranking? No — AI crawlers and search-engine crawlers are separate bots. Block the AI ones and leave Googlebot and Bingbot alone.

Google stopped indexing my site. Check whether a bot or block rule is catching it, and switch that rule to tag or dry run to confirm before changing anything else.

Real visitors keep getting challenged. Narrow the rule to the paths that actually need it (/login, /search) instead of the whole site.