If pages you care about are missing from Google, the cause is almost always a crawl or indexing error — and the place to diagnose it is the free tool most people still call Google Webmaster Tools. This guide is a practical walkthrough of how to fix crawl errors in Google Webmaster Tools: where the reports live now, what each error type means, and the exact steps to clear them so your content stays indexed and your traffic data stays clean.
First, the name change
"Google Webmaster Tools" was renamed Google Search Console back in 2015, and everything below lives inside it (search.google.com/search-console). People still search for the old name, so we use both here, but the product is Search Console. This matters for a second reason: the standalone "Crawl Errors" report from the old interface was retired in 2019. The information did not disappear — it moved and, honestly, got better. Google Search Console crawl errors are now surfaced across three places:
- The Pages report (indexing) — why URLs are or are not indexed.
- The URL Inspection tool — deep diagnosis of a single URL.
- The Crawl stats report (in Settings) — how Googlebot is fetching your site overall.
Learn those three and you can resolve any crawl problem the tool can show you.
Set up (or verify) your property
If you have not already, add and verify your site so Google will report on it:
- Open Search Console and click Add property. Choose a Domain property (covers every subdomain and both http/https via a DNS TXT record) or a URL-prefix property for a single origin.
- Verify ownership. The old "upload an HTML file to your root directory by FTP" method still works, but the easiest routes today are a DNS TXT record (for domain properties), a meta tag in your
<head>, or linking an existing Google Analytics or Tag Manager account that already has admin access. - Wait for data. Reports populate over the following day or two and then build history.
Verified owners get the full Pages report, URL Inspection, sitemap submission, and crawl stats.
Where crawl errors live now: the Pages report
In the left menu under Indexing → Pages, you get a summary of indexed vs. not-indexed URLs and, below it, a table of reasons pages are not indexed. This is the modern replacement for the old crawl-errors list. Click any reason to see the affected URLs. The reasons you will actually act on:
404 — Not found
Googlebot requested a URL that returns "not found." Fix depends on intent:
- The page should exist → restore it, or fix the broken internal/external link pointing at the wrong URL.
- The page moved → serve a 301 redirect from the old URL to the new one so link equity and any rankings carry over.
- The page is genuinely gone → that is fine. A 404 (or 410 "Gone") is the correct response for deleted content; you do not need to redirect everything. Just make sure nothing on your own site still links to it.
Server error (5xx)
Googlebot hit a 500/503-class response. This is a hosting or application problem, not an SEO one: a crashing script, a database timeout, an overloaded server, or a firewall throttling the bot. Check your server logs for the timestamp, fix the root cause, and confirm the URL loads reliably. Persistent 5xx errors make Google crawl less, so treat them as urgent.
Redirect error
A redirect chain that is too long, a loop, or a redirect to a bad URL. Collapse chains to a single hop (old URL → final URL directly) and remove loops.
Soft 404
The page returns HTTP 200 "OK" but looks empty or like an error page to Google ("no results found," a thin placeholder). Either add real content and make it worthwhile, or return a proper 404/410 if the page should not exist. Do not serve a 200 for what is really a missing page.
Blocked by robots.txt
You told crawlers to stay out. If that is intentional (admin areas, faceted-search junk), ignore it. If an important page is blocked by accident, edit robots.txt to remove the offending Disallow rule.
Excluded by "noindex"
The page carries a noindex meta tag or HTTP header. Remove it if the page should be indexed; keep it if the exclusion is deliberate.
Blocked due to access forbidden (403) / unauthorized (401)
Googlebot was refused. Usually a login wall, IP allowlist, or over-aggressive bot protection catching Googlebot. Make sure public pages are reachable without authentication and that your security layer is not blocking the real Googlebot.
Crawled — currently not indexed / Discovered — not indexed
Not an error exactly, but a quality/priority signal: Google found the URL and chose not to index it (yet), or fetched it and decided it was not worth indexing. The fix is rarely technical — improve the content's depth and uniqueness, strengthen internal links to the page, and make sure it is not a near-duplicate of something you already rank.
Diagnose a single URL: the URL Inspection tool
When you need to know exactly why one page is not indexed, paste it into the URL Inspection bar at the top of Search Console (this is the modern "webmaster tools crawl URL" workflow). It reports whether the URL is on Google, the last crawl date, the canonical Google chose, any indexing blocks, and mobile usability. Two buttons matter most:
- Test live URL — fetches the page right now, so you can confirm a fix works before asking Google to re-crawl. You can view the rendered HTML and the response code Googlebot receives.
- Request indexing — pushes the URL into the crawl queue after you have fixed it. Good for one-off, high-priority pages; not a bulk tool.
Confirm the fix: the Validation flow
After you correct a batch of errors, do not just wait and hope. In the Pages report, open the specific error reason and click Validate Fix. Google re-crawls the affected URLs and tracks the validation as Started → Passed (or Failed, if it still sees the problem). This gives you a clear, auditable signal that the issue is actually resolved rather than guessing from raw counts.
The big-picture view: Crawl stats
Under Settings → Crawl stats you get a report on how Googlebot fetches your whole site: total crawl requests over time, average response time, and a breakdown of responses by status code, file type, and purpose (discovery vs. refresh). Use it to spot systemic trouble the per-URL reports miss:
- A spike in 5xx or timeouts points at server capacity or throttling.
- A rising average response time slows crawling and, at scale, hurts how much of a large site Google indexes.
- A flood of requests to junk URLs (endless faceted-search or calendar parameters) wastes crawl budget that should go to real pages — block those patterns in
robots.txtor with parameter handling.
Keep sitemaps clean
Submit an XML sitemap under Indexing → Sitemaps and keep it honest: it should list only canonical, indexable, 200-status URLs — no redirects, no 404s, no noindex pages. A sitemap full of errors sends mixed signals and makes your crawl reports noisier than they need to be. Regenerate it whenever your URL structure changes. If you manage a large or frequently changing site, our guide on how to crawl a sitemap covers generating and validating one at scale.
Beyond crawl errors: content and traffic insight
The old Webmaster Tools was also where you checked which keywords brought organic traffic. That has moved and improved too. The Performance report (replacing the old "Search Queries" and "Content Keywords" views) shows, for every query and page, your impressions, clicks, click-through rate, and average position over any date range. This is your "before the click" data — how your content performs in search results:
- High impressions, low CTR on a query → your title and meta description are not compelling; rewrite them.
- Average position just below 10 (page two) → a strong candidate to push onto page one with better internal linking or a content refresh.
- Rising impressions on a topic cluster → real audience demand; consider expanding that content.
You can also connect Search Console to Google Analytics 4 so this search data sits alongside your on-site behavior data. Together they answer both halves of the question: how your pages perform in search, and what visitors do after they arrive.
FAQ
Where did the Crawl Errors report go? Google retired the standalone report in 2019. Its data now lives in the Pages (indexing) report, the URL Inspection tool, and Crawl stats in Settings.
Is Google Webmaster Tools the same as Search Console? Yes. It was renamed Google Search Console in 2015; the old name persists only in habit and search queries.
How do I ask Google to re-crawl a fixed URL? Use URL Inspection → Request indexing for a single page, or Validate Fix inside a Pages error group for a batch.
Do 404 errors hurt my rankings? A 404 for genuinely deleted content is normal and harmless. What hurts is important pages returning 404 by accident, or internal links pointing at dead URLs. Redirect moved pages; let truly gone pages 404.
How often should I check? Glance at the Pages report weekly and after any site change (migration, redesign, CMS update). Set aside time to act whenever the not-indexed count jumps.
Where scraping fits
Search Console is authoritative for your own site, but it stops at your property line. To understand rankings across competitors, track SERP positions for thousands of keywords, or audit a site you do not own, teams add web scraping for SEO on top of Search Console data. If you would rather have ranking, competitor, and crawl-health data collected and delivered on a schedule, scraping.pro provides it as data as a service — clean datasets you can drop straight into your reporting.