robots.txt in 2026: How to Control Googlebot, GPTBot, and ClaudeBot

Why Your robots.txt Needs to Evolve in 2026

In 2022, a well-configured robots.txt essentially meant: "Googlebot can access everything, except /admin/ and /checkout/." The landscape has changed dramatically. Since 2023, a new generation of AI crawlers has swept across the web: GPTBot (OpenAI, launched August 2023), ClaudeBot (Anthropic), PerplexityBot, CCBot (Common Crawl), ChatGPT-User, Applebot-Extended… and that's just the beginning.

These bots have very different purposes from Googlebot. Some train large language models (LLMs) on your content. Others power real-time responses in ChatGPT or Perplexity. Some build public web archives. The problem: they all respect robots.txt — but if you haven't explicitly configured them, your content is accessible to everyone by default.

In 2026, a professional robots.txt doesn't manage 3 bots. It manages 20+. And every decision has direct implications for your traditional SEO visibility and your presence in AI search engines (GEO).

The 6 Crawlers You Need to Know in 2026

Important clarifications:

• GPTBot is used to train future OpenAI models. Blocking it doesn't affect current ChatGPT responses, but influences future model versions.
• ChatGPT-User is distinct: it's used for ChatGPT's "Browsing" mode, which fetches URLs in real-time to answer users. Blocking it can reduce your visibility in current ChatGPT responses.
• ClaudeBot crawls for Anthropic's training data. Like GPTBot, its effect is on future models.
• PerplexityBot is directly tied to Perplexity.ai results — blocking it excludes you from Perplexity citations.
• CCBot (Common Crawl) feeds numerous open-source datasets used to train dozens of third-party AI models.

robots.txt Syntax: The Essential Rules

The robots.txt file follows the Robots Exclusion Protocol (REP). A few essential reminders before moving on to templates.

Basic structure

User-agent: [bot name]
Disallow: [blocked path]
Allow: [allowed path]
Crawl-delay: [seconds between requests]

Sitemap: [full sitemap URL]

Priority rules

1. Longest match wins: if you have Disallow: /blog/ and Allow: /blog/seo/, the path /blog/seo/article is allowed (longest rule wins).
2. Block order doesn't affect priority, but readability recommends placing priority bots first.
3. * is a wildcard in User-agent only: User-agent: * applies to all bots not explicitly listed.
4. An empty Disallow (Disallow:) means "everything is allowed" for that bot.
5. Comments are preceded by # and ignored by crawlers.

What robots.txt does NOT do

• It doesn't guarantee non-indexation (a page linked by another site can appear in Google even if blocked).
• It doesn't stop malicious bots (scrapers, non-compliant spiders) — for that you need rate-limiting or a WAF.
• It doesn't protect sensitive data (use authentication for that).

robots.txt Templates by Strategy

Template 1 — Allow everything (maximum SEO + maximum GEO)

If your goal is maximum visibility across all engines — Google, ChatGPT, Perplexity, etc. — this is the default template. Ideal for content publishers, blogs, and documentation sites.

# robots.txt — Maximum SEO + GEO visibility
# Updated: June 2026

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/
Disallow: /search?

Sitemap: https://your-site.com/sitemap.xml

With this template, Googlebot, GPTBot, ClaudeBot, and PerplexityBot all have access to your content. Your articles can be cited in ChatGPT, Perplexity, and used for training future models.

Template 2 — Block all AI/LLM crawlers

If you want to preserve your content for traditional SEO only, blocking LLM training and AI citations, use this template. Suitable for sites with premium content, proprietary data, or concerns about AI use of their content.

# robots.txt — Traditional SEO only, no AI crawling
# Updated: June 2026

# Classic search engines — allowed
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

# AI crawlers — blocked
User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: cohere-ai
Disallow: /

# All other unlisted bots — limited access
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /api/

Sitemap: https://your-site.com/sitemap.xml

Warning: blocking PerplexityBot excludes you from Perplexity.ai citations. Blocking ChatGPT-User reduces your visibility in real-time ChatGPT responses.

Template 3 — Selective authorization (targeted GEO strategy)

The most nuanced strategy: allow engines that generate traffic or visibility, block those that use your content only for training.

# robots.txt — Selective GEO strategy
# Updated: June 2026

# Classic SEO — fully allowed
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Real-time AI — allowed (citations in ChatGPT and Perplexity)
User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# LLM training — blocked (no direct return for you)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

# Default for all other bots
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Disallow: /api/
Disallow: /search?

Sitemap: https://your-site.com/sitemap.xml

This approach lets you appear in ChatGPT and Perplexity responses (which generate direct traffic) while protecting your content from passive AI model training with no return benefit for you.

SEO/GEO Implications of Each Choice

Blocking Googlebot

This is the catastrophic scenario: you disappear from Google SERPs. Never do User-agent: Googlebot / Disallow: / on a site that depends on SEO. Even a Disallow: /blog/ for Googlebot can significantly drop your organic traffic.

Blocking GPTBot

The impact is indirect and deferred. GPTBot crawls for training future OpenAI models. If you block it, you won't appear in future versions of ChatGPT/GPT (in responses generated from training data). The short-term impact is limited; the 2–3 year impact can be significant for your authority in the OpenAI ecosystem.

Blocking ChatGPT-User

Immediate impact: ChatGPT can no longer fetch your URLs in real-time during Browsing mode. If a user asks ChatGPT "summarize this article" pointing to your URL, they'll get an error or degraded response. Your site will be less likely to be cited in ChatGPT responses that require real-time verification.

Blocking PerplexityBot

Direct impact on your visibility in Perplexity.ai, one of the most widely used generative AI engines in 2026. If you target tech audiences or decision-makers, Perplexity is often their preferred search engine. Blocking PerplexityBot means accepting you won't be cited in its responses.

Blocking CCBot

CCBot feeds Common Crawl, a public dataset that serves as a training base for dozens of open-source models. The direct impact on your visibility is low in the short term, but it reduces your presence across the entire open-source LLM ecosystem.

The Big GEO Opportunity

GEO (Generative Engine Optimization) is the new discipline of optimizing your content to appear in AI engine responses. Visibility in ChatGPT, Perplexity, or Google AI Overviews is becoming as strategic as traditional Google rankings. Your robots.txt is the first line of control over your GEO presence.

robots.txt vs llms.txt: What's the Difference?

These two files are complementary, not interchangeable.

In practice: robots.txt decides whether a bot can access your content. llms.txt tells it what you want it to understand about your site — your key pages, your positioning, your expertise.

A well-optimized site for GEO in 2026 has both. robots.txt controls selective access to AI crawlers, and llms.txt guides LLMs to your best content.

For everything you need to know about llms.txt, read our complete llms.txt guide.

How to Test Your robots.txt

1. Google Search Console — robots.txt Tester

In Search Console, go to Settings > robots.txt Tester. You can verify that a specific URL is crawlable (or blocked) for Googlebot. This is the official tool, but it only tests Googlebot.

2. Manual testing with curl

To test how a specific bot would see your robots.txt, simulate its User-Agent:

# Test with GPTBot User-Agent
curl -A "GPTBot" https://your-site.com/

# View the robots.txt directly
curl https://your-site.com/robots.txt

# Check if a page is accessible for PerplexityBot
curl -A "PerplexityBot" -I https://your-site.com/blog/my-article/

3. Third-party tools

• Screaming Frog: full crawl with custom User-Agent simulation
• robots.txt Checker (various online tools): syntax validation
• Server logs: the most reliable method for knowing which bots actually crawl your site

4. How to see which bots are crawling your site

The best source of truth: your server logs. A filter on the User-Agent field of your Apache or Nginx logs will give you the complete list of bots visiting your site, their frequency, and the URLs they crawl.

# Extract User-Agents from Apache logs
grep -i "bot|crawler|spider" /var/log/apache2/access.log | awk '{print $12}' | sort | uniq -c | sort -rn | head -20

Recommended review frequency: every 3 months minimum. New AI crawlers appear regularly, and User-Agents can change.

FAQ — robots.txt and AI Crawlers

Does robots.txt really protect against non-compliant crawlers?

No. robots.txt is a trust-based convention — well-intentioned bots (Googlebot, GPTBot, ClaudeBot, PerplexityBot) respect it scrupulously because their reputation depends on it. But malicious scrapers, spam bots, or undeclared crawlers ignore it. For real protection against malicious bots, you need server-level rate-limiting, a WAF (Web Application Firewall), or solutions like Cloudflare Bot Management.

Can you block only certain sections of the site for AI?

Yes, that's exactly the advanced use of robots.txt. For example, you can allow GPTBot on your public blog but block access to premium content or client documentation:

User-agent: GPTBot
Allow: /blog/
Allow: /free-resources/
Disallow: /documentation/
Disallow: /client-area/
Disallow: /

How do you know which bots are crawling your site right now?

Three methods: (1) analyze your server logs filtering on User-Agents containing "bot", "crawler" or "spider"; (2) use Google Search Console to see Googlebot traffic; (3) set up alerts in your log analysis tool (Datadog, Loggly, etc.) for new unknown User-Agents.

Is robots.txt sufficient for GDPR compliance?

No. robots.txt has no legal value under GDPR. If personal data from your users is accessible via public URLs, blocking them in robots.txt doesn't protect you legally. For GDPR compliance, you need server-side authentication and access controls. robots.txt can be a good faith signal, but it's not legal protection.

Do you need a llms.txt file in addition to robots.txt?

In 2026, yes — if you have a GEO strategy. robots.txt controls access (who can crawl), but tells LLMs nothing about what matters on your site. llms.txt fills this gap: it guides AI engines to your key pages, explains your positioning, and maximizes your chances of being correctly cited in generated responses. Both files are complementary — robots.txt secures access, llms.txt optimizes understanding.

Conclusion

Your robots.txt is the first file read by every crawler visiting your site — search engines and AI alike. In 2026, leaving it in its 2022 state means leaving all doors open by default, with no strategy.

Three actions to take now:

1. Audit your current robots.txt: which bots are listed? Which are absent? Compare with the list of active crawlers in 2026.
2. Define your GEO strategy: do you want to appear in ChatGPT, Perplexity? Are you comfortable with your content being used for LLM training? Your answers determine your configuration.
3. Add a llms.txt to guide AI engines to your best content.

Want to know exactly how your robots.txt is configured and what it's letting through? Run a SeAudit audit — the report analyzes your robots.txt, your crawlability, and your GEO score in minutes.