About our crawler

If you found this page in your web server's logs, something at OSIC fetched a page from your site. This explains exactly what we are, what we take, and how to make us stop. You do not need to contact us first, and you do not need to justify the request.

How to identify us

Every request we make carries this User-Agent:

OSIC/0.1 (+https://osic.datatorturingsolutions.co.uk/crawling; abuse@datatorturingsolutions.co.uk)

Every request also carries an ORN — a scan identifier unique to the collection run that made it — in an OSIC-ORN request header:

OSIC-ORN: orn:osic:lon1:01JQ8Z...:scan/01JQ8Z...

It means you can match a specific line in your own access log to a specific run of ours, and quote it back to us, without needing our cooperation to do it. That is the whole point of it: our accountability to you should not depend on us being helpful.

What we collect

Publicly served web pages from company websites — the HTML your server returns to any visitor. We record page counts, titles, link structure, response headers and TLS certificate metadata. We do not log in, we do not submit forms, and we do not fetch anything behind an authentication boundary.
Public DNS records — resolved over DNS-over-HTTPS against a public resolver, which never touches your infrastructure at all.
Certificate Transparency logs — the public, append-only record of every TLS certificate issued for a domain. Again, nothing to do with your servers.
Public registries — WHOIS/RDAP, regional internet registry data, and (for UK companies) Companies House filings.

We do not port-scan, we do not probe for vulnerabilities, we do not attempt any input a normal browser would not send, and we do not try to find unlinked paths. Active scanning of third-party infrastructure is out of scope for this project by policy, not by omission.

How we behave

We read /robots.txt first, and we follow RFC 9309. A file that is served is obeyed. A file that is absent — a 404 — means no rules apply, which is what the standard says: most hosts on the internet have no robots.txt, and that is not a refusal. A server that is broken, or that will not answer at all, is treated as a refusal, because it has told us nothing.
Where we are stricter than the standard, deliberately. A robots.txt answering 401 or 403 is, strictly read, a 4xx and so permits crawling. We treat it as a refusal instead. Something looked at who we are and said no, and reading “you are forbidden” as “you may take anything” would be working around a block. A significant minority of the companies listed on this site are in exactly that position, and we leave them that way — the coverage page carries the current count.
We had this wrong until 2 September 2026. Every non-200 response was treated as a complete disallow, including a plain 404, which is not what the standard says and meant we declined to crawl sites that had simply never written a robots.txt. Fixed, and recorded here rather than quietly corrected.
We are rate limited — a token bucket, two requests per second per host by default, and lower wherever a site or a network has asked for it.
We stop when a site is unhappy. If the error rate over a rolling window exceeds a threshold, the run aborts rather than retrying harder.
We cap every run by page count and wall-clock time.
We come from stable, attributable addresses. We do not rotate IPs to get around a block. If you block us, we stay blocked — that is the arrangement, and rotating around it would destroy the only thing that makes us accountable to you.

How to stop us

Any one of these works, and none of them requires talking to us:

Disallow us in robots.txt. We honour it on the next run:
```
User-agent: OSIC
Disallow: /
```
Block our User-Agent or our address range at your edge. We will not work around it.
Email abuse@datatorturingsolutions.co.uk and ask. We will add you to our suppression list, which is checked on every write — not just at fetch time — so a removal survives subsequent collection runs rather than being undone by the next one.

If you want data about your organisation removed from this site as well as stopping future collection, say so in that email and we will treat it as a removal request, not just an opt-out. See our privacy policy for the legal basis we rely on and your rights under it.

Who we are

OSIC (OSINTCheck) is operated by Data Torturing Solutions Ltd. The public pages on this site are a free baseline dataset built from passive, publicly-available sources. What we have actually collected, and what we have deliberately not, is set out on the coverage page.