Block AI Training Crawlers Without Losing Your GEO Visibility

The Costly Reflex: Blocking GPTBot Without Understanding What It Does

Most site owners who add a "Disallow" line to their robots.txt to "block AI" end up blocking the wrong bot — or blocking everything, including the one bot that could actually cite their content in a ChatGPT or Perplexity answer. The confusion comes from a simple misunderstanding: "AI crawler" isn't one category. Since 2025, OpenAI, Anthropic, and Perplexity each run separate bots for two fundamentally different jobs — training their models, and indexing content in real time to answer live user questions. Blocking the first has zero effect on the second, and vice versa.

This article untangles the distinction, gives you a robots.txt template that blocks training without sacrificing your GEO visibility, and goes one step further with the TDM Reservation Protocol (TDMRep) — a lesser-known mechanism that leaves a record of your decision, beyond a plain robots.txt file.

GPTBot Is Not OAI-SearchBot: the Split Most robots.txt Files Miss

OpenAI runs three separate bots, each independently controllable in your robots.txt:

• GPTBot: crawls in bulk, without urgency, dedicated to training future models. Blocking it changes nothing about your presence in current ChatGPT answers.
• OAI-SearchBot: indexes continuously to power citations in ChatGPT Search. This is the one that matters if you want to show up as a source.
• ChatGPT-User: fires when a user pastes your URL directly into ChatGPT, or when an agent browses your site live in a user session.

Anthropic follows the exact same three-bot logic: ClaudeBot (training), Claude-SearchBot (indexing for Claude citations), Claude-User (live user requests). Perplexity runs two bots: PerplexityBot for its source pool, Perplexity-User for real-time retrieval. Google, for its part, uses Google-Extended — an opt-out token that doesn't crawl anything itself; it only changes how Google uses pages already crawled by Googlebot to train Gemini.

One more case worth knowing: Bytespider, ByteDance's bot, has a reputation for not always respecting robots.txt. Blocking it in the file isn't always enough — a block at the WAF or CDN level is sometimes needed if you actually want it out.

What It Actually Costs to Block Everything on Reflex

One number worth keeping in mind before cutting everything off: according to an analysis of AI crawler activity published in early 2026, training-bot traffic now accounts for roughly 82% of AI crawler activity on the web, up from 72% a year earlier — while AI-referred traffic to sites (citations that generate an actual human click) jumped 975% between January 2025 and January 2026. The share that matters for your GEO visibility is growing fast, but it's still a minority of the raw request volume hitting your server — which is exactly why the temptation to block everything "to save bandwidth" costs you the fastest-growing part.

Same gap on the crawl-to-referral ratio: a separate analysis puts Anthropic's crawl-to-referral ratio at roughly 70,900 requests per 1 human click generated, versus about 5 to 1 for Googlebot. That number explains why some admins block ClaudeBot on reflex — except when they also accidentally block Claude-SearchBot (both user-agents contain "Claude," and a sloppy rule often treats them as one block), they lose citations too, not just the training crawl that strains the server.

Concretely: if you run an e-commerce site with 50,000 product pages and your robots.txt has a single User-agent: * line followed by Disallow: /products/ to "protect the catalog from AI," you're blocking GPTBot AND OAI-SearchBot AND Claude-SearchBot indiscriminately — which amounts to voluntarily pulling yourself out of every ChatGPT or Claude answer about your products, when only the first one posed an actual IP concern.

A robots.txt That Blocks Training Without Sacrificing Citations

Here's a realistic starting point, to adapt to your own risk tolerance:

# Block model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow citation and search bots
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

Important note: this setup protects training, not the "live user-agent" crawl (ChatGPT-User, Claude-User, Perplexity-User), which fires when a real user interacts with your site through these AI tools — blocking them means stopping those users from viewing your content via their assistant, a different experience from automated crawling.

Going Beyond robots.txt: the TDM Reservation Protocol

robots.txt has been, since 1994, a courtesy agreement with no binding force: nothing technically stops a crawler from ignoring it. The TDM Reservation Protocol (TDMRep), a final report from the W3C community group, proposes something different: a machine-readable rights declaration explicitly referenced in EU law (Article 4 of Directive 2019/790 on copyright in the Digital Single Market, picked up again by AI Act 2024/1689). The protocol doesn't replace robots.txt — it adds a layer of evidence.

TDMRep can be declared three ways: a JSON file at /.well-known/tdmrep.json, an HTTP header sent by your server, or a meta tag in your pages' <head>. JSON file example:

{
  "tdm": [{
    "policy": "https://yoursite.com/tdm-policy",
    "tdm-reservation": 1
  }]
}

Setting tdm-reservation to 1 reserves your text-and-data-mining rights; 0 leaves them open. Worth noting, and often misunderstood: TDMRep has zero effect on your Google ranking or your citation by AI engines — the protocol only documents a training-rights position, it doesn't drive crawling, indexing, or ranking. In other words, you can reserve your training rights via TDMRep while keeping OAI-SearchBot and Claude-SearchBot wide open to keep getting cited — the two are independent. This remains a technical point, not legal advice: if the question carries real stakes for your business (premium content, proprietary data), the right move is to talk to someone qualified on the topic, not a blog post.

Which Choice Fits Your Profile

• SaaS doing content marketing, goal = visibility and leads: allow broadly. Your priority is getting cited, not protecting content that only has value once distributed. Blocking citation bots out of excess caution costs you more than what you think you're protecting.
• E-commerce with a proprietary catalog and pricing: allow citation bots on content pages (guides, comparisons), think case by case for product pages if pricing data carries real competitive value. If you want to see what a full audit looks like before diving in, check out a sample report.
• Media or publisher with premium/paid content: this is the profile where TDMRep makes the most sense as a complement to robots.txt — the machine-readable declaration creates a record in case of a dispute, which robots.txt alone doesn't.

Verifying Your Setup Actually Works

A well-written robots.txt guarantees nothing if the bot ignores it, or if a proxy/CDN silently rewrites your rules. The only reliable check runs through server logs: search for the actual user-agents in raw requests, confirm GPTBot no longer shows up after blocking it and that OAI-SearchBot still gets through. Our server log analysis guide for AI bots walks through the method request by request. For full robots.txt syntax, including managing Googlebot alongside AI bots, our 2026 robots.txt guide covers the whole thing.

FAQ

Does blocking GPTBot in my robots.txt cut me out of ChatGPT? No. GPTBot is only used for training future models. Your presence in ChatGPT Search answers depends on OAI-SearchBot, a completely separate bot you can keep open even while blocking GPTBot.

Does the TDM Reservation Protocol replace robots.txt? No, it adds to it. robots.txt blocks technical access; TDMRep declares a position on usage rights, with an explicit legal basis in EU law, even if a crawler accesses the page anyway.

How do I know if my robots.txt is actually being respected? By analyzing your server logs to spot the real user-agents behind incoming requests, not just trusting the file itself — some bots, like Bytespider, have a reputation for not always honoring it.

Key Takeaways

An AI crawler isn't one category: OpenAI, Anthropic, and Perplexity explicitly separate their training bots (GPTBot, ClaudeBot) from their citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking the former without touching the latter protects your content from training without sacrificing your GEO visibility. The TDM Reservation Protocol goes further by adding a machine-readable rights declaration, complementary to robots.txt.

Want to check where your current setup is quietly leaking GEO visibility? Get your score out of 100 with a SeAudit audit.