HTML analysis

ID: cia-2010-covert-communication-websites/html-analysis

CIA 2010 covert communication websites / HTML analysis by Ciro Santilli 37 Updated +Created
The HTML from the index page of Wayback Machine were:
The best way to analyse the HTML is to grap our dumps from: github.com/cirosantilli/cia-2010-websites-dump.
Some possibly interesting searches include:
  • list all HTML comments, maybe something spicy was left over:
    git grep '<!--'
  • search for weird file extensions:
    git ls-files | grep -Ev '\.(jpg|gif|html|txt|png|css|php|js|jar|cgi|htm|swf|ico|JPG|class|zip|sf)'
  • have a look at the largest folers:
    ncdu
Some of the HTML files contain conditional comments e.g. web.archive.org/web/20091023041107/http://aquaswimming.com/ contains:
<!--[if IE 6]> <link href="swimstyleie6.css" rel="stylesheet" type="text/css"> <![endif]-->
Varios of the non-English websites seem to have comments translating the content e.g.:
./noticiasmusica.net/20101230165001/index.html:<h2>Alguns dos Melhores Sites Nacionais</h2><!--some of the best national sites (in music)-->
This feels like it could be the translation helping the technical webdev team know what is what.
Many of the RSS frame pages use:
<base target="_blank" />
which is a weird HTML tag that would lead all links to open on new tabs, e.g. web.archive.org/web/20110202124411/http://thecricketfan.com/home.html.
Various websites have pages with .php extension. It feels likely that all websites were written in PHP.
Some URLs existed both in HTML and .php extension, or were converted at some point:
allworldstatistics.com/20110207151941/comprehensivesources.html
allworldstatistics.com/20130818155225/comprehensivesources.php
A few of the PHP urls have weird IDs in them like omktf, juqwt and qlaqft:
./middle-east-newstoday.com/20100829004127/omktf/uirl.php?ok=461128
./newsandsportscentral.com/20100327130237/juqwt/eubcek.php?pe=747155
./pondernews.net/20100826031745/lldwg/qlaqft.php?fc=281298
we wonder what they mean.
A few separate websites have an archive with the same pid parameter:
fightwithoutrules.com/20131220205811/?pid=2POQ7BC1G/index.html
half-court.net/20131223165013/?pid=2POQ7BC1G/index.html
health-men-today.com/20131223002237/?pid=2POQ7BC1G/index.html
intlnewsdaily.com/20131221121441/?pid=2POQ7BC1G/index.html
intoworldnews.com/20131217193621/?pid=2POQ7BC1G/index.html
It is unclear what it means. All of them contain something like:
<html>
<head>
<meta name="robots" content="noarchive" />
<meta name="googlebot" content="nosnippet" />
</head>
<body>
<div align=center>
<h3>Error. Page cannot be displayed. Please contact your service provider for more details.  (11)</h3>
</div>
</body>
</html>
so looks like an archival artifact only.
The following two websites have a feeds.php system for their RSS:
./mydailynewsreport.com/20110211111053/myrss/feeds.php?src=http:/www.refahemelli.com/pashto/news/rss.php&chan=y&desc=1&targ=y&utf=y
./magneticfieldnews.com/20110208063545/magneticfeeds/feeds.php?src=http:/www.bbc.co.uk/pashto/index.xml&chan=y&desc=1&targ=y&utf=y
Some of the HTML uses attributes without quotes, which is legal, but very unusual nowadays:
soldiersofsouthasia.com/20110207203705/home.htm: <a href=http://www.rss-to-javascript.com
We can try to search for any link leaks by listing all domains linked to with:
git grep --no-color -I -h --no-line -o 'https?://[^/">?]+[/">?]' | sed -r 's/.$//' | sort | uniq -c | sort -nk1
The first thing that shows up is that there are some IPs linked to directly! But they seem to be the direct IPs of legitimate websites, we are not sure why IPs were used rather than domain names:
We can also get the full line for each with sorted by least common domains with the slow:
git grep --no-color -I -h --no-line -o 'https?://[^/">?]+[/">?]' | sed -r 's/.$//' | sort | uniq -c | sort -nk1 | awk '{if ($1 < 10) print $2}' | xargs -I{} git --no-pager grep -h --no-line -o '{}.*<' | tee tmp.log
We can search for all IP-like strings with:
git grep '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\b'

New to topics? Read the docs here!