A page can be perfectly crawled, well-linked, and fast, then vanish from Google because one misconfigured tag told it not to index. Confusing robots.txt, the meta robots tag, and the X-Robots-Tag HTTP header is one of the most common causes of accidental de-indexing we see when auditing SaaS and e-commerce sites. This guide clears up the three mechanisms, lists the directives that actually matter in 2026, breaks down the #1 trap that makes pages disappear without anyone understanding why, and answers the GEO question: do GPTBot and ClaudeBot follow the same rules as Googlebot?
Meta robots, X-Robots-Tag, robots.txt: three tools, one constant mix-up
Three mechanisms, three levels of control:
| Mechanism | Where | Controls | Works on |
|---|---|---|---|
| robots.txt | Root file | Crawling (bot access) | Entire site |
| Meta robots tag | <head> HTML | Indexing and SERP display | HTML pages only |
| X-Robots-Tag | HTTP header | Indexing and SERP display | HTML + non-HTML files (PDF, images, JSON) |
The part most sites get wrong: robots.txt blocks bot access, but it does not guarantee a URL stays out of the index. If external links point to a Disallow'd page, Google can still index it, without a snippet, just the bare URL. To actually pull a page out of results you need the noindex directive, and for Google to see it, the bot has to be able to crawl the page first. The two mechanics conflict if you don't sequence them correctly.
The directives that matter in 2026
The meta robots tag and the X-Robots-Tag header share the same directive vocabulary, combinable with commas:
| Directive | Effect |
|---|---|
index / noindex | Allow / block indexing (default: index) |
follow / nofollow | Allow / block following every link on the page (default: follow) |
noarchive | Prevents a visible cached copy in results |
nosnippet | Hides the text and video snippet in results, also blocks citation in AI Overviews |
max-snippet:[n] | Limits the snippet to n characters (-1 = unlimited) |
max-image-preview:none|standard|large | Controls image preview size |
max-video-preview:[n] | Limits video preview length in seconds |
noimageindex | Blocks indexing of images present on the page |
notranslate | Stops Google from offering an automatic translation |
unavailable_after:[date] | Automatically de-indexes the page after a given date |
The meta robots tag: syntax and implementation
In raw HTML, the tag goes in the <head>:
<meta name="robots" content="noindex, follow" />
On Next.js (App Router), it's declared directly in page metadata, versioned alongside the rest of the code:
export const metadata = {
robots: { index: false, follow: true },
};
On WordPress, Yoast SEO and Rank Math expose a simple "Allow search engines to show this page in search results" toggle per post, category, or content type. On Shopify or Webflow, control happens through the page's native SEO settings, no need to hand-edit the tag.
X-Robots-Tag: when the HTML tag isn't enough
A PDF, an image, or a JSON file has no <head>. There's nowhere to place a meta robots tag. For these resources, the only option is the X-Robots-Tag HTTP header, sent by the server before the content is even loaded.
Nginx:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}
Apache (.htaccess):
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
Next.js (next.config):
async headers() {
return [{ source: "/(.*\.pdf)", headers: [{ key: "X-Robots-Tag", value: "noindex" }] }];
}
A real case from an audit: an e-commerce site exporting its product sheets as PDFs for customer support ended up with over 4,000 PDFs indexed by Google, duplicating product page content and diluting topical authority. A single X-Robots-Tag noindex on /exports/*.pdf cleaned up the index within three weeks, without touching a line of HTML or losing traffic on the product pages themselves. The freed-up crawl budget even sped up Googlebot's pass on newly added sheets.
Trap #1: Disallow + noindex, the combo that breaks everything
This is the most common and most counter-intuitive mistake on the topic. The disaster sequence:
- A thin-content or outdated page is still live on the site
- You add
noindexto its<head>to pull it from the index - You also add the page to robots.txt's Disallow, "just to be safe"
- Result: Googlebot can no longer crawl the page, so it never sees the
noindex, so the page stays indexed, sometimes for months, showing a bare "No information is available for this page" in the SERP
The rule: never both directives on the same URL at once. Correct sequence:
- Confirm robots.txt allows crawling the page
- Add
noindex(tag or header) - Wait for de-indexing, confirmed in Search Console → URL Inspection → "Excluded by 'noindex' tag"
- Only then, if you also want to save crawl budget on that URL, add the Disallow
Across SeAudit audits, this conflict shows up on a significant share of the e-commerce and SaaS sites checked, almost always on filter pages, internal search result pages, or old campaign landing pages never cleaned up.
The GEO angle: do AI crawlers respect your directives?
GPTBot, ClaudeBot, PerplexityBot, and Google-Extended all state they respect robots.txt for crawling. But the classic meta robots tag was not originally built for these engines, it targets Googlebot and traditional search engines. In practice in 2026:
- Blocking an AI crawler happens via robots.txt (
User-agent: GPTBotthenDisallow: /), not via meta robots nosnippethas a documented effect from Google: a page carrying this directive cannot appear as a source in AI Overviews, even while staying normally indexed in classic results- There is currently no standardized directive equivalent to
noindexthat specifically targets AI engines outside of AI Overviews, the real GEO control lever remains robots.txt scoped by user-agent
If your GEO strategy targets citations in ChatGPT or Perplexity, check that you don't have a stray nosnippet on your pillar pages: it's one of the most silent blockers found in audits, often inherited from a misconfigured SEO plugin, and has nothing to do with a content problem.
JavaScript pages: watch out for noindex added too late
On a client-rendered site (React without SSR, some SPA setups), if noindex is injected after the component mounts, Googlebot reads the raw HTML on the first pass, then waits for a second rendering pass to see DOM changes, a delay that can range from several days to several weeks. Result: a page meant to disappear from the index keeps showing up in results for weeks. The only reliable option remains server-side rendering (SSR/SSG), which includes the tag directly in the initial HTML response.
How to check your directives in 5 minutes
- Meta robots tag:
curl -s https://yoursite.com/page | grep -i 'name="robots"' - X-Robots-Tag header:
curl -sI https://yoursite.com/file.pdf | grep -i x-robots-tag - Actual indexing status: Search Console → URL Inspection → "Coverage" tab
- Automated overview: SeAudit's audit scans noindex, nofollow, X-Robots-Tag, and robots.txt conflicts across the entire site in one pass, with per-page detail in the full report
Key takeaways
- robots.txt controls crawling, meta robots and X-Robots-Tag control indexing: they are not synonyms
- Never Disallow + noindex on the same page at once: Google must crawl the page to see the noindex
- X-Robots-Tag is the only option for de-indexing non-HTML files (PDF, images, JSON)
nosnippetblocks citation in AI Overviews, even while the page stays normally indexed- On client-side JS, only SSR/SSG guarantees the noindex is seen on Googlebot's first pass
FAQ
What's the difference between robots.txt and the noindex tag?
robots.txt prevents crawling, the bot never accesses the page. The noindex tag prevents indexing, but assumes the bot can first crawl the page to read it. Using both together on the same URL prevents Google from ever seeing the noindex, which can leave the page indexed indefinitely.
How do I block a PDF from being indexed?
A PDF has no HTML <head>, so no meta robots tag is possible. The only solution is the X-Robots-Tag HTTP header, configured at the server level (Nginx, Apache, or a Next.js middleware) to return X-Robots-Tag: noindex for that file type.
Does noindex take effect immediately?
No. Google has to re-crawl the page to read the new directive, then de-index it. This delay typically ranges from a few days to several weeks depending on the page's crawl frequency. To speed things up, request re-indexing via Search Console after adding the noindex.
Does nofollow on a page block every link it contains?
Yes, nofollow set at the meta robots tag level applies to every link on the page, unlike rel="nofollow" set on an individual link, which only affects that one link.
Do AI crawlers like GPTBot respect the noindex tag?
Their documented behavior mostly concerns robots.txt, which they all state they respect for crawling. The classic noindex tag remains primarily designed for traditional search engines. To specifically control AI crawler access, robots.txt with per-user-agent rules (GPTBot, ClaudeBot, PerplexityBot) remains the most reliable lever.
A robots.txt/noindex conflict can make dozens of pages disappear without you knowing why. Run your free SeAudit audit to catch these conflicts across your entire site, or upgrade to the full report for a page-by-page prioritized fix plan.