Verified bots

How Botect recognizes legitimate search and AI crawlers and allows them by default — so good bots are never blocked for looking automated.

Not every bot is hostile. Search engines, AI crawlers, uptime monitors, and link previewers are bots you want visiting your site. Botect maintains a verified-bot allowlist so these are recognized and allowed — never caught by enforcement meant for abusive automation.

How verification works

A session is marked verified_bot only on actual verification — never on a user-agent claim alone (anyone can send Googlebot in their UA). When a session matches the allowlist, the combiner short-circuits:

verified_bot → true
verified_bot_category → the bot's category (e.g. Search, AI Crawler, AI Search)
band → verified
bot banding is skipped entirely

Allowing verified bots

The allow_verified toggle is on by default. With it on, any verified bot resolves to allow as the very first step of verdict resolution — before rules and other toggles. This is almost always what you want.

{ "allow_verified": true }

Verified-bot handling runs first. A block_definite toggle or a low-score rule won't catch a verified crawler while allow_verified is on.

Acting on verified bots

If you do want to treat some verified bots differently — say, challenge AI crawlers on a paid section — you have two levers:

Turn allow_verified off to stop the blanket short-circuit, then let rules and toggles decide.
Write a rule that matches on verified_bot / verified_bot_category:

verified_bot AND verified_bot_category == "AI Crawler" AND path == "/premium"
→ block

When allow_verified is on, your enforcement rules should include AND NOT verified_bot so a broad rule never accidentally targets a crawler you rely on for indexing.

Using it in rules

verified_bot (bool) and verified_bot_category (string) are both rule-usable signal fields:

NOT verified_bot AND score < 30        → block        # bots only, never crawlers
verified_bot_category in ["Search"]    → allow         # always let search engines through

The verified-bot list is maintained from public crawler registries and refreshed periodically, so newly-announced legitimate crawlers are recognized without any change on your side.

Was this page helpful?