Crawl budget: why it is probably not your problem

Crawl budget is a real concept, and for the overwhelming majority of sites it is a distraction. If you have a few hundred or a few thousand pages, search engines are not rationing their visits to you, and no amount of optimisation will make your content get indexed faster.

The term is worth understanding mainly so you can recognise the situations where it does apply, and stop worrying about all the others.

What it actually means

A crawler cannot visit every URL on the web as often as it would like, so it decides how much attention each site gets. Two things drive that: how much it believes your site is worth revisiting, and how much your server can take without struggling.

Notice that neither is a quota you can top up. A site that publishes valuable things regularly gets crawled more, because that is worth doing, not because it earned points. It is a consequence, not a lever, and that misreading produces most of the pointless optimisation on this subject.

Are you concerned? The table that settles it

The first two rows cover very nearly every site that asks the question. If you are in them, the best decision is to close this article and go write a useful page. That is not a rhetorical flourish: the hours spent on crawl directives by sites in those two rows are hours that produced nothing measurable, and they were taken from something that would have.

Who this really concerns

Sites with an unlimited URL space. A shop where every combination of filters is a distinct address. A calendar that will happily generate the year 2147. A search page that turns any query into a crawlable URL. These do not have ten thousand pages, they have an infinity, and that is the actual problem: it is not an insufficient budget, it is bottomless spending.

Very large catalogues: hundreds of thousands of real pages, where the tail genuinely competes for attention. At that scale the question stops being technical and becomes editorial: which pages deserve to exist.

Sites that respond slowly. If your server takes two seconds per page, a crawler backs off to avoid hurting you. Here the fix is not an SEO setting: it is the response time, and it also benefits your visitors, which makes it the only item on this list that pays twice.

What people mistake for a crawl budget problem

A pile of 404s. They look wasteful, and they are not. A crawler learns that a URL is gone and stops asking. Fix the ones something still points at, not the rest, and above all do not delete pages to "lighten" anything.

Pages that are not indexed. Almost always a quality or duplication judgement, not a budget one. The page was seen and not kept, which is a different conversation, and it is settled by improving the page rather than by tuning a crawler.

A slow launch after a migration. New URLs get discovered progressively. That is normal and has nothing to do with rationing: the migration check-list covers what actually helps, namely clean redirects and a current sitemap.

Redirect chains. They do cost requests, but the problem they create is elsewhere: they end up being cut, and they lose visitors long before they cost any crawling. The single-hop rule justifies itself without invoking crawl budget at all.

If you genuinely are in scope

The work is about removing addresses, not adding directives.

Stop generating URL combinations nobody should see: that is a template setting, not a config file. Use robots.txt for the infinite spaces, calendars and internal search first among them. Return proper status codes so a crawler knows what to forget. Keep the sitemap limited to pages you actually want visited, and strip out anything returning an error.

A word on noindex versus robots.txt, which are constantly confused: the first asks for a page not to be indexed although it will still be visited, the second asks for it not to be visited at all. For an infinite space you want the second, and it is also the one that makes a page permanently invisible, so it is handled with care.

Before writing a rule into robots.txt, test it: an over-broad directive closes off a whole section of the site without announcing anything. Checking what a crawler is allowed to visit takes ten seconds and avoids the most expensive mistake on this page.

Where the worry comes from, and why it persists

The subject has a history, and knowing it helps you drop it. The phrase comes from documentation aimed at very large sites, where it described a real problem of resource allocation. It was then picked up by general-purpose guides that dropped the size condition, and what remained was a worry with no threshold.

It persists for two reasons. First because it is comfortable: it offers a technical explanation for a lack of results, which is easier to accept than the editorial one. Second because most people cannot test it, having no access to their server logs. A belief that cannot be tested does not die.

The test that settles it is one question: is a page I published last week indexed? If it is, your site is crawled as much as it needs to be, and the subject is closed.

What genuinely helps you get crawled

Three things, none of them called a budget.

Internal links to your deep pages. A page no link reaches from your homepage within three clicks is a page crawlers find late and revisit rarely. It is the most effective lever, and the only one that also serves your readers.

A clean sitemap. Current, no error URLs, no blocked pages, honest modification dates. A sitemap that lies about dates quickly stops being taken seriously.

A site that answers quickly. That is a hosting matter rather than an SEO one, and it benefits the people reading you first. Broken links and cascading redirects contribute too, which is one more reason to deal with them without ever invoking crawl budget.

How to measure instead of guessing

Your server logs are the only place this question can be answered honestly. They say which URLs were requested, by which crawler, how often, and with what status code.

Three numbers are enough, taken before and again a month later:

• The number of distinct URLs crawled. If it is far above your real page count, you have an infinite space somewhere.
• The share of error responses. A high proportion indicates a structural problem, not a budget one.
• The share of requests hitting pages of no interest. It is the only measurement that justifies doing anything.

Without those three numbers, any crawl budget optimisation is a belief. With them, the question is usually settled in an afternoon, and most of the time the conclusion is that there was nothing to do.

One job does deserve your time and has nothing to do with crawlers: the links leaving your site for pages that died elsewhere. Those appear in no crawl report, and your readers meet them first.