You've allowed GPTBot and ClaudeBot in your robots.txt, you even added a clean llms.txt, and yet your site never shows up in ChatGPT or Perplexity answers. The culprit is almost always in the same place, and it isn't your SEO: it's your CDN. Cloudflare (and the others) can block an AI crawler before it even reads your robots.txt, silently, with a 403 or a JavaScript challenge page you'll never see while browsing normally.
This guide covers the layer most SEO articles skip: server/CDN-level blocking. It's a blind spot precisely because it's invisible from a normal browser, you see your site perfectly, so you assume everyone does. A crawler gets turned away at the door.
robots.txt says "yes", your CDN says "no"
Two unrelated layers you must separate:
- robots.txt is a polite request. It tells the bot "I'd rather you didn't go there". A well-behaved bot obeys; that's the topic of our robots.txt guide for Googlebot, GPTBot and ClaudeBot.
- The CDN's firewall / bot detection is a lock. It decides who gets a response at all. A bot blocked here never reads your content, not even your robots.txt.
In other words: you can have the friendliest robots.txt in the world, but if Cloudflare serves a challenge to GPTBot, your page is invisible to ChatGPT. The two settings contradict each other constantly, unnoticed, because they live in two different dashboards often managed by two different people.
The three Cloudflare settings that cost you AI citations
1. Bot Fight Mode (and Super Bot Fight Mode)
The most common trap. Cloudflare's Bot Fight Mode, a one-click toggle in the Security tab, sends a challenge (JavaScript computation or a "Just a moment…" page) to any traffic it deems automated. The problem: its definition of "automated" regularly catches legitimate crawlers, including some AI bots and, in aggressive setups, subsets of Googlebot itself.
Super Bot Fight Mode (paid plans) is finer-grained: it lets you handle "verified bots" separately. But its default of "Block" or "Managed Challenge" for unverified bots is exactly what snags emerging AI crawlers, which aren't all on Cloudflare's verified-bot list yet.
What to do: if you care about GEO visibility, turn off plain Bot Fight Mode, and under Super Bot Fight Mode set "verified bots" to Allow. Then create explicit WAF rules to allow the user-agents of the AI crawlers you want exposure to (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot).
2. Overly broad WAF and managed rules
A web application firewall (WAF) rule written to block "bots" or a whole country also catches the data centers AI crawlers originate from (often US cloud IPs). A rule like "challenge anything from this ASN" can shut the door on Perplexity without you meaning to. Review your custom rules and managed rulesets: look for anything that challenges or blocks on IP-reputation or ASN criteria.
3. Crawler Hints: the false friend
Crawler Hints starts from a good intention: notify crawlers when your content changes to avoid wasteful crawls. But adoption varies by bot, and on rarely-updated sites, combined with aggressive caching, it can make some crawlers believe nothing moved and slow their re-crawl. It's not a hard blocker like Bot Fight Mode, but on a site where freshness matters (and for AI engines, it does), it's a setting to watch rather than enable reflexively.
How to know if you're blocked (the 2-minute test)
The symptom is invisible while browsing but trivial to test from the command line. Impersonate an AI crawler and check the status code:
curl -A "GPTBot" -I https://your-site.com
curl -A "ClaudeBot" -I https://your-site.com
curl -A "PerplexityBot" -I https://your-site.com
What you want: a HTTP/2 200. What should alarm you:
- 403 Forbidden → a WAF rule or Bot Fight Mode is blocking that user-agent.
- 503 / "Just a moment…" page in the body → a Cloudflare JavaScript challenge. An AI crawler won't run it and leaves empty-handed.
- 429 Too Many Requests → rate-limiting too aggressive, punishing a crawler that reads several pages in a row.
Compare with a normal browser user-agent (curl -A "Mozilla/5.0..."): if the browser gets through and GPTBot gets a 403, you've found your culprit. This is exactly the kind of gap an automated audit detects, the SeAudit score raises a "site behind an anti-bot challenge" signal when our own crawler gets challenged, precisely because AI engines hit the same wall.
The GEO angle: a CDN block is exclusion, not demotion
In classic SEO, a technical error pushes you down the results. In GEO it's blunter: if an AI engine's crawler can't fetch your page, you're not ranked poorly, you don't exist in its corpus. There's no "page 2" of AI answers. Either your source is readable and citable, or it's absent.
That's what makes this setting disproportionately important relative to the effort it takes: unblocking AI crawlers at the CDN level takes ten minutes and can be the difference between "never cited" and "cited". Conversely, leaving Bot Fight Mode on by default quietly sabotages your generative visibility, the worst class of problem, because no report warns you about it.
The "AI crawler allowed at the CDN level" checklist
- Test your user-agents with the three
curlcommands above (GPTBot, ClaudeBot, PerplexityBot) and aim for 200 everywhere. - Disable plain Bot Fight Mode; under Super Bot Fight Mode, set verified bots to Allow.
- Add an explicit WAF allow rule for the AI-crawler user-agents you want to serve.
- Review custom / managed rules: hunt any challenge or block based on ASN, country or IP reputation that might catch cloud IPs.
- Check rate-limiting: a crawler reads several pages fast; too low a threshold cuts it off.
- Keep robots.txt consistent with what you allow at the CDN, both layers must say the same thing (see also our robots.txt and security / HTTPS / headers guides).
The principle to remember: your AI-visibility strategy is only as good as its weakest link. Perfect content, flawless schema and a lovingly crafted llms.txt are worthless if the front door is locked at the CDN. Start by making sure the crawler gets in, everything else comes after.