We had our robots.txt rule wrong, and it excluded sites that had done nothing

This site has made a point of its crawler being conservative about robots.txt. On 2 September 2026 we found that it was not being conservative. It was being wrong, in a way that quietly excluded sites that had done nothing but decline to write a file they were never obliged to write.

What the standard says

RFC 9309 is the 2022 standardisation of the robots exclusion protocol, and section 2.3.1 is unambiguous about each class of response:

• 2xx — parse the file and obey it.
• 3xx — follow the redirect.
• 4xx — “unavailable”. The crawler may access any resource. Most hosts on the internet have no robots.txt; that is not a refusal, and never has been.
• 5xx — “unreachable”. The crawler should assume a complete disallow. A broken server has not told you anything.

We had collapsed the third and fourth cases together. “Fail closed” sounds like the cautious choice and in the 5xx case it is. Applied to a 404 it is not caution, it is a misreading: it treats the absence of a rule as the presence of a prohibition.

What it cost

Two things, one small and one not.

The small one: our crawler follows images to the content delivery networks that serve them, and CDNs very often have no robots.txt at all. In the first run of our image-metadata collection, 23 of 25 images were refused on that basis. After the correction, the same sample fetched 40 of 40.

The larger one: the same function decided whether a company had a reachable site at all. A company whose robots.txt merely 404s was recorded as having no working origin and was never collected against. That is a company excluded from this site's analyses for the offence of not having written an optional file.

Where we are still stricter than the standard, on purpose

A 401 or 403 is also a 4xx, and the letter of the RFC permits crawling after one. We continue to treat both as refusals.

The difference is that those are not an absence of rules. Something looked at who we are and said no. Reading “you are forbidden” as “you may take anything” is exactly the sort of manoeuvre this project has said it will not perform, and a standard that permits it does not oblige us to.

That distinction is why the correction does not change the headline number in our earlier piece on companies whose CDNs block us. Those sites answer the request with a 403 aimed at our user agent. They are refusing, not silent, and they stay excluded. We checked before assuming: of 40 excluded companies sampled, 36 serve robots.txt with a 200 and exactly one returned a 404.

Which means most of our exclusions are still unexplained

That check was meant to size the damage and instead produced a better question. If 36 of 40 excluded companies serve a perfectly good robots.txt, the robots rule was never why they were excluded. Something else is, and we do not yet know what — the current suspicion is sites that render entirely in the browser, where a crawler that reads bytes rather than running JavaScript finds an empty shell and records nothing.

That is roughly 119 companies, and it is now the largest known gap in this corpus. It is stated here rather than left in a backlog because the coverage page exists to make gaps visible, and an unexplained gap is still a gap.

Why publish this

A crawler's behaviour is a claim it makes about itself, and this site had that claim written on two of its own pages. Both have been corrected. The alternative — fixing the code and quietly editing the copy — would have left every previously published figure resting on a description of our method that was no longer true, with no way for a reader to tell which side of the change a number came from.

Everything on this site collected before 2 September 2026 was gathered under the stricter rule. That makes those figures conservative rather than wrong: we crawled less than we were entitled to, never more.

See about our crawler for the current rules in full.