The Hundred Greatest Theorems by Paul and Jack Abad (1999) Updated 2025-07-16
View more
Read the full article
Amazon AI accelerator silicon Updated 2025-07-16
View more
Read the full article
Metabolomics Updated 2025-07-16
View more
Study of the metabolome.
Read the full article
Microplate Updated 2025-07-16
Read the full article
Tesla (unit) Updated 2025-07-16
Read the full article
Zone of Avoidance Updated 2025-07-16
Read the full article
Common Crawl Updated 2025-07-16
View more
Amazing project, that basically makes a more searchable Wayback Machine.
A bit hard to use their data though, partly due to size, but also lack of free to use querrying mechanisms, and how obtuse Amazon S3 is to use.
Notably, aws-cli with an account is the only reliable way, everything else is way too broken, e.g. trying the to check the an index index.commoncrawl.org/CC-MAIN-2023-06/ very often 500s.
But still, their projct is amazing.
The only out-of-the-box search they seem to have is: urlsearch.commoncrawl.org/ for domains/URLs. It is good, but there could be so much more... notably IPs.
Also could should document the data shape a bit better.
To explore the data, after login:
aws s3 ls s3://commoncrawl/crawl-data/CC-MAIN-2013-20/
Copy the toplevel directory only:
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/ . --recursive --exclude "*/*"
Copy some wet/wat files:
aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz .
aws s3 sync s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz .
Directory structrure:
  • cc-index.paths.gz (1K)
  • cc-index-table.paths.gz (1K)
  • segment.paths.gz (1.7K) Sample lines:
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/
crawl-data/CC-MAIN-2013-20/segments/1368696381630/
  • index.html (2.3K)
  • wat.paths.gz (98K) Sample lines:
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wat.gz
  • wet.paths.gz (98K) Sample lines:
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wet.gz
  • warc.paths.gz (99K)
    crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.gz
  • segments: directgory with actual data
    • 1368696381249: one of many segments, any meaning of name?
      • CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz (142M, 334M unzipped)
        A tiny bit of metadata, and then plaintext content from the website, e.g. the second one:
        WARC/1.0
WARC-Type: conversion
WARC-Target-URI: http://004eeb5.netsolhost.com/stephensilver.htm
WARC-Date: 2013-05-18T08:11:02Z
WARC-Record-ID: <urn:uuid:773b31ba-ddc6-47a5-ae24-d08141b9944d>
WARC-Refers-To: <urn:uuid:4b1bdbff-4926-4ced-86f6-072f5bb3837a>
WARC-Block-Digest: sha1:LQFSCR2LIJQYMPTXRHWU7HAPQTVSYS3A
Content-Type: text/plain
Content-Length: 12046

Stephen Silver is a journalist and editor who specializes in the areas of politics, pop culture, film and sports. He works as an editor with the North American Publishing Co. and as a film critic with The Trend, a local newspaper in the Philadelphia area.
        No IP unfortunately.
      • CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz (329M, 1.4G unzipped)
        A lot of JSON metadata and no contents as desired. Contains IP! Some entries however are humongous with a ton of useless data, that's what bloats these so much:
        WARC/1.0
WARC-Type: metadata
WARC-Target-URI: CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
WARC-Date: 2013-11-22T14:51:12Z
WARC-Record-ID: <urn:uuid:ec54e493-8965-41be-b344-07596cc30b3a>
WARC-Refers-To: <urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>
Content-Type: application/json
Content-Length: 1180

{"Envelope":{"Format":"WARC","WARC-Header-Length":"274","Block-Digest":"sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR","Actual-Content-Length":"372","WARC-Header-Metadata":{"WARC-Type":"warcinfo","WARC-Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz","WARC-Date":"2013-11-22T14:51:12Z","Content-Length":"372","WARC-Record-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Type":"application/warc-fields"},"Payload-Metadata":{"Trailing-Slop-Length":"0","Actual-Content-Type":"application/warc-fields","Actual-Content-Length":"372","Headers-Corrupt":true,"WARC-Info-Metadata":{"robots":"classic","software":"Nutch 1.6 (CC)/CC WarcExport 1.0","description":"Wide crawl of the web with URLs provided by Blekko for Spring 2013","hostname":"ip-10-60-113-184.ec2.internal","format":"WARC File Format 1.0","isPartOf":"CC-MAIN-2013-20","operator":"CommonCrawl Admin","publisher":"CommonCrawl"}}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"453","Header-Length":"10","Inflated-CRC":"866052549","Inflated-Length":"650"},"Offset":"0","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}

WARC/1.0
WARC-Type: metadata
WARC-Target-URI: http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions
WARC-Date: 2013-05-18T05:48:54Z
WARC-Record-ID: <urn:uuid:d519658f-7a63-46c1-849b-4cd92332ddb8>
WARC-Refers-To: <urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>
Content-Type: application/json
Content-Length: 1501

{"Envelope":{"Format":"WARC","WARC-Header-Length":"433","Block-Digest":"sha1:B2B6JDSGWCUQIIUGV54SXEE25RX4SANS","Actual-Content-Length":"302","WARC-Header-Metadata":{"WARC-Type":"request","WARC-Date":"2013-05-18T05:48:54Z","WARC-Warcinfo-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Length":"302","WARC-Record-ID":"<urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>","WARC-Target-URI":"http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions","WARC-IP-Address":"165.1.125.44","Content-Type":"application/http; msgtype=request"},"Payload-Metadata":{"Trailing-Slop-Length":"4","HTTP-Request-Metadata":{"Headers":{"Accept-Language":"en-us,en-gb,en;q=0.7,*;q=0.3","Host":"ap.org","Accept-Encoding":"x-gzip, gzip, deflate","User-Agent":"CCBot/2.0","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},"Headers-Length":"300","Entity-Length":"0","Entity-Trailing-Slop-Bytes":"0","Request-Message":{"Method":"GET","Version":"HTTP/1.0","Path":"/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions"},"Entity-Digest":"sha1:3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ"},"Actual-Content-Type":"application/http; msgtype=request"}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"455","Header-Length":"10","Inflated-CRC":"453539965","Inflated-Length":"739"},"Offset":"453","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}
        Let's beautify one of them to see it better:
        
{
  "Envelope": {
    "Format": "WARC",
    "WARC-Header-Length": "274",
    "Block-Digest": "sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR",
    "Actual-Content-Length": "372",
    "WARC-Header-Metadata": {
      "WARC-Type": "warcinfo",
      "WARC-Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz",
      "WARC-Date": "2013-11-22T14:51:12Z",
      "Content-Length": "372",
      "WARC-Record-ID": "<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>",
      "Content-Type": "application/warc-fields"
    },
    "Payload-Metadata": {
      "Trailing-Slop-Length": "0",
      "Actual-Content-Type": "application/warc-fields",
      "Actual-Content-Length": "372",
      "Headers-Corrupt": true,
      "WARC-Info-Metadata": {
        "robots": "classic",
        "software": "Nutch 1.6 (CC)/CC WarcExport 1.0",
        "description": "Wide crawl of the web with URLs provided by Blekko for Spring 2013",
        "hostname": "ip-10-60-113-184.ec2.internal",
        "format": "WARC File Format 1.0",
        "isPartOf": "CC-MAIN-2013-20",
        "operator": "CommonCrawl Admin",
        "publisher": "CommonCrawl"
      }
    }
  },
  "Container": {
    "Compressed": true,
    "Gzip-Metadata": {
      "Footer-Length": "8",
      "Deflate-Length": "453",
      "Header-Length": "10",
      "Inflated-CRC": "866052549",
      "Inflated-Length": "650"
    },
    "Offset": "0",
    "Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"
  }
}
        Fuck no IP addresses either. But other entries do have it, why not this one?
        The reason these can be huge is the HTML-Metadata section which contain all outlinks! gist.github.com/Smerity/e750f0ef0ab9aa366558#file-bbc-pretty-wat-L34
      • CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz ()
        Obtain:
        aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz .
Read the full article
Computational problem Updated 2025-07-16
View more
Read the full article
Fandom (website) Updated 2025-07-16
Read the full article
TransferWise Updated 2025-07-16
View more
This is a good company, first they truly helped reduce international transfer fees. They they continued to morph into a decent challenger bank.
Their Wise Interest account was amazing as of late 2023: wise.com/gb/interest/
Instant access with representative national interests and 0.29% fees.
Brick and mortar banks were way way behind in that regard!
E.g. October 2023, Wise was doing 4.87% interest after fees, while Barclay's best option was 1.16% above 5k pounds on the Rainy Day Saver (5% below). Ridiculous!
Read the full article
Transversal time dilation Updated 2025-07-16
View more
Light watch transverse to direction of motion. This case is interesting because it separates length contraction from time dilation completely.
Of course, as usual in special relativity, calling something "time dilation" leads us to mind boggling ideas of "symmetry breaking": if both frames have a light watch, how can both possibly observe the other to be time dilated?
And the answer to this, is the usual: in special relativity time and space are interwoven in a fucked up way, everything is just a spacetime event.
In this case, there are three spacetime events of interest: both clocks start at same position, your beam hits up at x=0, moving frame hits up at x>0.
Those two mentioned events are spacelike-separated events, and therefore even though they seem simultaneous to you, they are not going to be simultaneous to the moving observer!
If little clock one meter away from you tells you that at the time of some event (your light beam hit up) the moving light watch was only 50% up, this is just a number given by your one meter away watch!
Read the full article
Amazon anticompetitive acquisition Updated 2025-07-16
View more
Amazon is apparently notorious for having bought off many competitors, many of them just to kill off the competition and clear the way, not to actually reuse them.
youtu.be/tfAhTtBlb2Q?t=849 from Video "Jeff Bezos Revealed by Bloomberg (2015)" clearly shows Tim O'Reilly saying that very clearly about Bezos.
I do know of a number of cases in which he [Bezos] has acquired companies in order to take out competitors, potential future competitors. Rather than because he actually wants that business to continue.
Perhaps O'Reilly who is the bookselling business is not the greatest fan of Jeff. But still. My God.
www.yalelawjournal.org/pdf/e.710.Khan.805_zuvfyyeh.pdf Amazon's Antitrust Paradox by Lina M . Khan from The Yale Law Journal raises this incredible issue.
Read the full article
Computer manufacturer Updated 2025-07-16
View more
This section is about companies that integrate parts and software from various other companies to make up fully working computer systems.
Read the full article
Computer science course of the University of Oxford Updated 2025-07-16
View more
Course lists: www.cs.ox.ac.uk/teaching/courses/ True to form, courses appear to have identifiers, e.g. qi for the Quantum Information course of the University of Oxford rather than more arbitrary A1/A2/A3, B1/B2/B3, naming convention used by the Mathematics course of the University of Oxford and the Physics course of the University of Oxford, and URLs can either have years or not:
The "course materials" section of each course leads to courses.cs.ox.ac.uk/ which is paywalled by IP (accessible via Eduroam): TODO which system does it use? Some courses place their materials directly on "www.cs.ox.ac.uk", and when that is the case they are publicly accessible. So it is very much hit and miss. E.g. www.cs.ox.ac.uk/teaching/courses/2022-2023/quantum/index.html from Quantum Processes and Computation course of the University of Oxford has the assignments such as www.cs.ox.ac.uk/people/aleks.kissinger/courses/qpc2022/assignment1.pdf publicly visible, but e.g. www.cs.ox.ac.uk/teaching/courses/2022-2023/modelsofcomputation/ has nothing.
Handbook:
Read the full article
Condenser microphone Updated 2025-07-16
Read the full article
Cuisine by region Updated 2025-07-16
Read the full article
Cycling lobbying entity Updated 2025-07-16
Read the full article
Data breach Updated 2025-07-16
Read the full article
List of fighting game Updated 2025-07-16
Read the full article
Puzzle Updated 2025-07-16
Read the full article

There are unlisted articles, also show them or only show them.