Every site and every server we host blocks bad bots and fake crawlers by default. You do not need to switch anything on, and it costs nothing extra.
What is blocked?
- Known bad bots: scrapers, content thieves, spam tools and vulnerability scanners that announce themselves by name.
- Fake crawlers: programs that pretend to be a well-known crawler, such as Googlebot, Bingbot, Applebot, GPTBot or ClaudeBot, to get past your defences. A request that claims to be one of these but does not come from that company's own published network addresses is refused.
The check happens twice: on our edge network before a request reaches your server, and again on the web server itself, so a request that goes straight to your server is checked too.
What is never blocked?
Real, verified crawlers always get through. Googlebot, Bingbot, Applebot and the real AI crawlers (such as GPTBot and ClaudeBot) are recognised by where they come from, not just by the name they give, so your site stays in search results and in AI answers.
Can I block AI crawlers on my site?
Yes, and the choice is yours for each site. Open Sites, choose the site, open the CDN tab and find Security and bot control → AI crawlers:
- No special treatment: AI crawlers are treated like any other visitor.
- Allow them: AI crawlers are let through.
- Challenge them or Block them: AI crawlers are challenged or refused at the edge. These two options come with the bot protection included in your plan; the screen shows if yours does not include it.
We never change this setting for you. Blocking fake bots does not touch it: a real AI crawler is only blocked if you choose to block it.
Something legitimate was refused
Uptime monitors, payment callbacks and SEO tools are not affected unless they pretend to be one of the crawlers above. If a service you use is refused, open a support ticket with the time and the address it came from and we will look at it.