A quarter of the biggest listed companies can't tell us what they allow

Since 1994, the arrangement has been simple. A crawler asks for /robots.txt, the site says what it allows, and the crawler obeys. It has no legal force; it works because both sides find it useful. It is the closest thing the web has to a machine-readable statement of consent.

OSIC takes it literally. Our crawler fetches /robots.txt before anything else, and if it cannot read one, it treats the entire site as disallowed rather than allowed. The comment in our own source is blunt about why: an absent or broken robots.txt is not consent. Fail open and you have quietly decided that silence means yes.

That rule produced a result we did not expect.

The file that says what you may do is the file you may not read

The list is not a list of small or obscure companies. It includes AstraZeneca, BP, BT Group, Tesco, Unilever, Aviva, Sainsbury’s, Compass Group and Imperial Brands from the FTSE 100, and Tesla, Oracle, Mastercard, McDonald’s, Merck, Morgan Stanley, Caterpillar and Berkshire Hathaway from the S&P 500.

Ask any of them for /robots.txt and you do not get a policy. You get this:

HTTP/2 403

<H1>Access Denied</H1>
You don't have permission to access
"http://www.tesco.com/robots.txt" on this server.
Reference #18.d81e1202...

That is Akamai. AstraZeneca’s is CloudFront, with the same outcome in different wording. The bot-management layer in front of the site has decided the request is not worth serving — and because the thing it declined to serve is the consent file itself, a well-behaved crawler is left with no way to find out what it was allowed to do. It has to assume the answer is nothing.

There is a circularity here worth stating plainly. The mechanism exists so that sites can express a preference to automated clients. The bot-management layer blocks automated clients before they can read the preference. A crawler that ignores robots.txt entirely is unaffected by any of this. The only crawler that is affected is the one that was trying to comply.

What actually triggers the block

Our first assumption was that this was about our address — that a datacentre IP with no reputation was being filtered. That turned out to be wrong. The same requests fail identically from our collection host and from an ordinary UK residential connection.

It is the User-Agent. We ran the same request against the same five hosts, back to back, varying only the User-Agent string, and repeated the whole exercise from both networks. The results were identical on both:

Read the first two columns together, because that is the finding. A bare, anonymous curl/8.x is served the file. Our User-Agent — which names the product, links to a page explaining what we collect, and carries an abuse contact address — is refused by every one of them.

The Googlebot column is the control that explains the rule. A request claiming to be Googlebot from an address that plainly is not Google’s is refused too — correctly, because that is how you catch impersonation. These systems classify by declared identity: known-good crawlers from the right networks are allowed, impersonators are blocked, and anything identifying itself as a crawler nobody has heard of is blocked alongside them. There is no route on that list for a new, honest, small crawler. You cannot become known-good, because the mechanism by which you would demonstrate good behaviour is the one being denied to you.

What we are not going to do about it

The workaround is obvious and we are not taking it. Sending a browser User-Agent would clear several of these hosts today. It would also be a lie told to a machine that is specifically asking who we are, and it would make every promise on our crawler page unverifiable — the point of putting a User-Agent and a scan identifier on every request is that a site operator can hold us to them. A crawler that disguises itself cannot be held to anything. Rotating addresses to evade a block is out for the same reason.

So these 45 companies stay uncrawled, and we have changed how this site describes them. They used to be labelled “not yet crawled”, which implied a backlog that would eventually clear. They are now labelled crawling declined, which is what actually happened: we asked, the answer came back unreadable, and we stopped.

Their company pages are not empty, because most of what OSIC publishes never required fetching their site at all. DNS records, Certificate Transparency logs, WHOIS and Companies House filings are all served by third parties, and none of them are affected. You can map a company’s infrastructure in real detail without ever sending it a packet — which is the whole premise of passive collection.

The part that should worry a site operator

Every company above is paying for a bot-management product, and that product is almost certainly doing what it was bought to do. But somebody chose a configuration in which the request that says tell me your rules so I can follow them is answered with a 403, and it is worth asking whether that was a deliberate decision or just the default.

The scrapers these systems exist to stop do not read robots.txt. They were never going to. The only clients that request it are the ones that intended to comply — so blocking it filters out exclusively the well-behaved, while the badly-behaved carry on unaffected. If you run one of these sites, serving /robots.txt to everyone, unconditionally, costs you nothing and restores the only channel you have for telling automated clients what you want. It is a static text file. It is meant to be read.

Method: every result above was re-verified live from two separate networks immediately before publication, rather than read from our stored crawl data. Status codes are for https://<host>/robots.txt with no redirects followed. The 45-company figure is the count of published companies whose most recent crawl aborted because neither the apex nor the www hostname returned a readable robots.txt; the current number is always on the coverage page. One host in our original sample (bt.com) has since started serving robots.txt to us again, which is a reminder that these configurations change without notice.