A common and frustrating SEO problem: you publish pages, they are good, and weeks later Google still has not indexed half of them. Search Console shows them as “discovered but not indexed” or “crawled but not indexed”. The content is fine. The problem is often crawl budget, one of the least understood parts of technical SEO.
What crawl budget actually is
Crawl budget is the number of pages Googlebot will crawl on a site within a given timeframe. It is a function of two things: crawl rate limit (how fast Google can crawl without overloading the server) and crawl demand (how much Google wants to crawl the site, based on its perceived importance and freshness).
For a small site of a few hundred pages, crawl budget is rarely a concern. For a large site, an ecommerce store with thousands of products, or any site generating lots of URL variations, crawl budget becomes a real constraint. Google simply does not crawl every page often enough.
The biggest crawl budget wasters
Faceted navigation. Filter and sort parameters generating thousands of near-duplicate URLs. Googlebot crawls them all, exhausting the budget on URLs that should never be indexed.
Redirect chains. Every hop in a redirect chain is a crawl. A chain of three redirects costs three times the crawl budget of a direct link.
Soft 404s and error pages. Pages that return a 200 status but show no real content waste crawl budget and confuse Google about site quality.
Infinite spaces. Calendars, paginated archives with no end, session ID parameters. These generate effectively unlimited low-value URLs for Googlebot to wander through.
Slow server response. If the server is slow, Google crawls fewer pages per session to avoid overloading it. Site speed is a crawl budget factor, not just a ranking factor.
How to reclaim crawl budget
The strategy is to point Googlebot at the pages that matter and away from the pages that do not. Use robots.txt to block crawling of parameter-based URLs that should never be indexed. Use canonical tags to consolidate duplicate URLs. Fix redirect chains so every redirect goes straight to the final destination. Return proper 404 or 410 status codes for genuinely dead pages. Keep the XML sitemap clean, listing only the pages you want crawled and indexed.
Internal linking also directs crawl budget. Pages that are well-linked internally get crawled more. Orphan pages get crawled rarely or never. A strong internal linking structure naturally guides Googlebot to the important pages.
How to diagnose a crawl budget problem
The Crawl Stats report in Search Console shows how many pages Google crawls per day, the average response time, and the breakdown of what is being crawled. If a large share of crawl activity is going to parameter URLs, redirects, or error pages, that is budget being wasted.
The Index Coverage report shows the gap between submitted and indexed pages. A large “discovered but not indexed” count on a big site is a strong crawl budget signal.
When crawl budget is not the problem
Sometimes “crawled but not indexed” is not a budget problem at all. It is a quality problem. Google crawled the page, assessed it, and decided it was not worth indexing. If a small site with no crawl budget constraints has pages that will not index, the answer is usually to improve those pages, not to fiddle with crawl directives.
Distinguishing a crawl budget problem from a content quality problem is the first diagnostic step, and getting it wrong wastes effort on the wrong fix.
Who needs to care
Crawl budget is mostly a concern for large sites: ecommerce stores, news sites, large service businesses, anything with thousands of URLs. Small business brochure sites rarely need to think about it. But for the sites that do have a crawl budget problem, it is often the single biggest reason good content is not getting indexed.
Crawl budget optimisation is firmly in technical SEO territory and takes someone who can read crawl logs and Search Console data accurately. Sydney SEO Partner’s technical SEO team handles crawl budget management as part of technical auditing. If Google is not indexing your pages and you do not know why, a free SEO audit will diagnose whether it is a crawl budget problem or something else.
