D'oh.

But to be serious. The </Wayback Machine> contains a very large proportion of all sites. It is the most complete database we have found so far. Some archives are very broken. But those are rares.

The only problem with the </Wayback Machine> is that there is no known efficient way to query its archives across domains. You have to have a domain in hand for CDX queries: <Wayback Machine CDX scanning>.

The <Common Crawl> project attempts in part to address this lack of querriability, but we haven't managed to extract any hits from it.

CDX + <2013 DNS Census> + heuristics however has been fruitful however.


Cia 2010 covert communication websites/wayback machine

ID: cia-2010-covert-communication-websites/wayback-machine