Ciro Santilli All IP ranges have some holes in them for which we don't have a domain name.
It is because there was nothing there, or just because we don't have a good enough reverse IP database?
It can't be HTML crawl because presumably there wouldn't have been links to those websites? Presumably this is why Common Crawl doesn't seem to have any hits.
So they must have had some kind of DNS A record database?
Or would IPv4 sweep have worked, without the
Host header with the CIA's setup?The same question also applies to the 2013 DNS Census. It has less hits, but still has many.
Whatever they did, we are so so glad that they did!
.com and .net are very dominant. Here we list other choices made:
.info: has a few hits:Did a full Wayback Machine CDX scanning on .info after:That makes about 10k domains, so it's about the right size.grep -e news -e noticias -e nouvelles -e world -e global.org: has a least one hit, see: Are there .org hits?.biz:- unarchived comms:
- atthemovies.biz
- unarchived comms:
Previously it was unclear if there were any .org hits, until we found the first one with clear comms: web.archive.org/web/20110624203548/http://awfaoi.org/hand.jar
Later on, two more clear ones were found with expired domain trackers:further settling their existence. Later on newimages.org also came to light.
- azerinews.org
- autism-news.org
Others that had been previously found in IP ranges but without clear comms:
- 65.61.127.177: material-science.org
- 212.4.17.61: tech-stop.org
Others in IP ranges by unarchived:
- 74.116.72.244 arborstribune.org
.org is very rare, and has been excluded from some of our search heuristics. That was a shame, but likely not much was missed.