A "Size 42" Filter Can Generate 3,800 URLs Nobody Asked For
Open Search Console, go to the Pages section, and compare the number of crawled URLs to your actual number of product categories. On a store with poorly scoped faceted navigation, the gap runs into the thousands of parasite pages — filter combinations (color, size, price, material, availability) that no human and no AI ever asked for, but that Googlebot crawls anyway.
On the Google side, this problem hasn't changed in 2026: wasted crawl budget, authority diluted across dozens of variants of the same page, thin pages accidentally indexed. What has changed is the extra exposure on the AI answer-engine side — GPTBot, ClaudeBot, PerplexityBot — which have neither the budget nor the rendering capability of Googlebot to sort through this noise, with a direct impact on which page they end up citing (or not).
Facets, Pagination, Sorting: Three Mechanics, Three Different SEO Treatments
These three get mixed up constantly, even though they don't cause the same problem:
- Facets (filters):
?color=red&size=42, combinable infinitely. This is the number-one source of duplicate content, because the number of possible URLs explodes combinatorially. - Pagination:
?page=2,?page=3. Not duplicate content in the strict sense, but poor linking between pages can break product discovery further down the list. - Sorting:
?sort=price-asc. Changes the display order, never the content. Block it from crawling systematically, no exceptions: no search value ever justifies letting Google or an AI crawler explore a differently-sorted page.
Sorting is the easy case. The real subject is facets.
The Concrete Problem: Duplicate Content and Runaway Crawl Budget
A case seen in audit: a ready-to-wear store with 600 active products and 6 filters (color, size, price, material, brand, availability) generated just over 14,000 crawlable URLs once every facet combination was accounted for — 23 parasite URLs for every product actually for sale. Of those 14,000 URLs, fewer than 30 matched a query with any measurable search volume (mostly brand + category combinations). The rest existed only because the filter engine generated a crawlable <a href> link for every checkbox.
Measured result before the fix: Googlebot spent roughly 40% of its weekly crawl budget on these filter variants, versus 60% on actual product and category pages. After switching to AJAX filtering with no crawlable links and adding targeted Disallow rules, the share of budget spent on real pages rose to 91% within six weeks, with a visible acceleration in the indexing of new products.
The 4 Methods to Rein In Facets (and When to Use Each)
| Method | Effect | When to Use |
|---|---|---|
| Canonical to the parent category | Consolidates the ranking signal without blocking the crawl | Low-value combinations but reasonable crawl volume |
| Noindex (meta robots or X-Robots-Tag) | Removes the page from the index; Google keeps crawling it | Combinations occasionally useful to users but with no search demand |
| Disallow in robots.txt | Blocks crawling at the source | Combinations with very high URL volume and low value (e.g. Disallow: /*?*sort=) |
AJAX with no <a href> link | The filter changes the display client-side; no crawlable URL is ever generated | Recommended default in 2026 for any new filter engine |
The classic trap: combining noindex AND disallow on the same URLs. If Google isn't allowed to crawl a page, it can never discover the noindex tag inside it — the page can stay indexed (URL only, no snippet) for months. Pick one method per combination category, never both at once.
One thing that still trips up teams in 2026: Search Console's old "URL Parameters" report was retired back in 2022. There's no declarative setting left where you can tell Google "ignore this parameter." The only remaining lever is the implementation itself — canonical, noindex, disallow, or AJAX with no link — applied directly on the site. That's also why server log analysis (see below) matters more than ever: it's the only reliable way to know what Google is actually crawling, now that the dedicated report is gone.
The GEO Angle: AI Crawlers Handle Facets Even Worse Than Googlebot
GPTBot, ClaudeBot, and PerplexityBot don't execute JavaScript. An AJAX filter engine with no <a href> link — the solution recommended above — is therefore completely invisible to them: they'll never see the filtered content, only the base category page. That's good news for crawl cleanliness, but it also means your "clean" category page alone has to carry all the content that would let an AI cite you for a specific query ("best leather sneakers under $80," for instance): it will never see the equivalent filtered version.
Second difference: based on the log analyses we run in audits, these bots operate with much tighter crawl envelopes than Googlebot on most sites. A site leaking tens of thousands of facet URLs to these crawlers mechanically reduces the odds they reach your high-value product and category pages before exhausting their session budget. In practice: the same fixes that protect your Googlebot budget (AJAX with no links, targeted Disallow rules) protect your AI-engine exposure even more, because their tolerance margin for noise is lower.
Finally, if a facet combination has real search value ("red dress size 42" gets volume), give it a clean, indexable URL with dedicated editorial content (description, reviews, FAQ) rather than relying on the raw filter — that's the content, and only that content, a generative AI engine will be able to rephrase and cite.
On multilingual sites, facets combined with hreflang make the problem worse still: every filter parameter multiplied by every language can blow up the number of declared alternates, with a real risk of hreflang reciprocity errors that Google — and AI engines, which also rely on this signal to decide which version to cite to a given user — penalize. Better to exclude faceted URLs from your sitemap and hreflang tags entirely, and only declare parent categories and validated high-demand combinations.
Quick Audit Checklist
- List the active URL parameters on your filter engine (Search Console > Pages, or a crawl with Screaming Frog sorted by parameters)
- Check that no combination stacks both noindex and disallow
- Confirm that sorting (
?sort=) is systematically blocked from crawling - Identify filter + category combinations with real search volume and give them a clean, indexable URL
- Analyze your server logs to compare crawl budget spent on facets versus real product/category pages
- Confirm that no crawlable
<a href>link is generated for low-value filters
Key Takeaways
Faceted navigation isn't a problem you solve once and forget: it's an ongoing trade-off between three URL families — the ones to block (sorting, no-volume facets), the ones to consolidate (canonical to the parent category), and the ones to index cleanly (high-demand combinations). Since AI engines have neither JavaScript rendering nor Googlebot's patience, poorly scoped facets cost you twice: in classic crawl budget, and in your odds of being cited by an AI on your most precise commercial queries.
Want to know if your filter engine is leaking crawl budget and diluting your AI visibility? Run a SeAudit audit and get your score out of 100, check a sample full report to see the level of detail before diving in, or dig deeper into crawl budget in our crawl budget and indexation guide and our canonical tags and duplicate content guide.