You check your rankings, your traffic, maybe even your AI citations. But have you ever opened the "Crawl Stats" report in Search Console? It's probably the least-visited tab in the entire tool, yet it's the one place where Google literally tells you: here's how Googlebot sees your site, how often it shows up, and where things break down before it even shows up as an indexing problem. Here's how to read it correctly, what it doesn't show, and a five-minute test you can run today.
What the report actually shows
The report (Search Console > Settings > Crawl Stats) shows three time series over a rolling 90 days: total crawl requests (every fetch of a URL by Google, successful or not), total bytes downloaded, and your server's average response time. These aren't vanity metrics — a sharp drop in request volume paired with a spike in response time is often the first visible signal of a server problem, weeks before it shows up as lost rankings.
Below those charts, four breakdowns explain the why: by response code, by file type, by purpose (Discovery vs Refresh), and by Googlebot type.
Reading the response code breakdown
This section groups requests into 2xx, 3xx (redirects), 4xx (client errors, including 404s), and 5xx (server errors). A high 4xx rate isn't automatically a problem — a site with a fast-turnover product catalog (e-commerce, marketplace) will mechanically generate 404s on discontinued products. What should worry you is a rising share of 5xx: that means Googlebot is hitting errors on your side, not pages that legitimately no longer exist. Every request that fails with a 5xx is a wasted request that could have crawled a page that actually matters — the direct link to your crawl budget is immediate.
Discovery vs Refresh: the signal almost nobody reads
This is the most misunderstood breakdown in the report. Google distinguishes two crawl purposes: Discovery, when the URL has never been crawled before, and Refresh, when it's a recrawl of an already-known page. On a brand-new site, a high Discovery ratio is normal and healthy. The problem shows up on a mature site: if you've been publishing for over a year and the report still shows a majority of Discovery requests over Refresh, it means a meaningful chunk of your URLs still aren't "settled" in Google's index — usually because they're poorly linked from your internal structure, or orphaned. This symptom is rarely fixed on the crawl side; it's fixed by going back to the site's structure.
Worth noting too: a sudden Discovery spike on an already-mature site isn't automatically good news either. It usually means you just shipped a large batch of new URLs — a category expansion, a faceted navigation rollout, a migration — and Google is now spending crawl budget figuring out whether they're worth keeping in the index. If that spike coincides with a drop in Refresh requests on your existing pages, you're effectively trading crawl attention on content that already ranks for crawl attention on content that hasn't proven itself yet. That trade-off is sometimes intentional, but it should never be accidental.
By Googlebot type: what the Smartphone vs Desktop split reveals
The report also breaks requests down by agent: Smartphone, Desktop, Image, Video, Page resource (JS/CSS), and specialized bots like AdsBot or StoreBot. With mobile-first indexing standard for years now, the Smartphone bot should account for the overwhelming majority of HTML requests on a typical site. If it's only a minority share, that's a technical signal worth taking seriously — especially if your site serves different rendering or content on mobile, in which case Google is literally indexing the version it visits the least.
One useful detail if you run Dynamic Search Ads: AdsBot recrawls your target URLs roughly every two weeks, independently of regular Googlebot. An isolated spike on that line has nothing to do with an SEO indexing issue — it's just Google Ads refreshing its dynamic ads.
Host status: the three checks that can stop everything
At the top of the report, the "Hosts" section groups three health indicators independent of your pages' content: robots.txt fetch success, DNS resolution, and server connectivity. Google marks a critical threshold with a dotted red line on each chart — if your curve crosses it, Google has hit enough failures to consider your host unreliable, which can slow down or temporarily suspend crawling of the entire site. It's the most basic check in the report, yet the one that most often explains an "unexplained" crawl drop: a robots.txt that returns a server error (instead of a clean 404) can block crawling of the whole domain, not just that file.
The report's blind spot: it doesn't see AI crawlers
Here's something Google's own documentation never states outright, but a plain reading of the labels confirms: this report only covers the Google crawler family (Googlebot, Image/Video bots, AdsBot, StoreBot). It includes neither Google-Extended (Gemini model training), nor GPTBot, nor ClaudeBot, nor PerplexityBot. If your goal is to know whether AI answer engines actually visit your site to cite you, this report will tell you nothing — zero rows, zero data. For that, you either need to cross-reference raw server logs directly (see our server log analysis guide), or at minimum verify that your robots.txt actually allows the right AI crawlers instead of accidentally blocking them. Plenty of sites get their Crawl Stats report to zero errors while remaining completely invisible to ChatGPT or Perplexity: these are two separate projects, with two separate data sources.
The five-minute test
Open your Crawl Stats report right now and do this: switch to "By purpose" and note the Refresh percentage. If your site is over a year old and that number sits below 50%, it's a sign a meaningful share of your pages remains in a grey zone for Google. Then switch to "By Googlebot type": if Smartphone doesn't clearly dominate Desktop, check whether your mobile rendering hides content compared to the desktop version. Both checks take less time than reading this paragraph, and both point to concrete actions rather than guesswork.
When this report isn't enough
The report stays aggregated and works off a sample of URLs that is, in Google's own words, "not comprehensive" — it's built to spot trends and anomalies, not to audit one specific URL. If you need to diagnose why one particular page isn't indexed, URL Inspection is the right tool, not Crawl Stats. And if your main concern is visibility in AI engines rather than classic Google crawling, this report will never be your starting point.
Key takeaways
- Crawl Stats shows three metrics (requests, volume, response time) and four breakdowns (response, file type, purpose, bot type) over a rolling 90 days.
- A high Discovery ratio on a mature site signals an internal linking problem, not a crawl problem per se.
- The Smartphone vs Desktop split should reflect mobile-first indexing: an imbalance is a real technical signal.
- The Host status section (robots.txt, DNS, connectivity) is the first place to check after an unexplained crawl drop.
- This report covers no AI crawler at all (Google-Extended, GPTBot, ClaudeBot, PerplexityBot): for GEO, you need to look elsewhere.
Want to know where your site stands on these points without digging through the report yourself? Run your free audit and get your SEO and GEO score out of 100 in a few minutes.