New articles - OurBigBook.com

Because JavaScript is a relatively crap/ad-hoc language, it ended up some decent tooling to make up for that, e.g. stuff like linting via ESLint and reformatting through Prettier is much more widespread than in other languages.

JavaScript data structure are also quite a bit anemic, which makes libraries such as lodash incredibly popular. But most of that stuff should be in the stdlib.

Our JavaScript examples can be found at:

Node.js example: examples that don't interact with any browser feature. We are just testing those on the CLI which is much more convenient.
JavaScript browser example: examples that interact with browser-specific features, notably the DOM

 Read the full article

Client-side storage by

Ciro Santilli 40 Updated 2025-07-16

 View more

www.google.com/search?q=client-side+storage&oq=Client-side+storage&aqs=chrome.0.0l3j0i22i30l4j69i60.88j0j7&client=ubuntu&sourceid=chrome&ie=UTF-8

 Read the full article

Clear client-side storage by

Ciro Santilli 40 Updated 2025-07-16

 Read the full article

Clear client-side storage on Chromium by

Ciro Santilli 40 Updated 2025-07-16

 View more

 Read the full article

Phaser.js by

Ciro Santilli 40 Updated 2025-07-16

 View more

Likely the best JavaScript 2D game engine as of 2023.

github.com/photonstorm/phaser
phaser.io/
phaser.io/examples/v3.85.0/games contains the demo games

Uses Matter.js as a physics engine if enabled. There's also an alternative (in-house?) "arcade" engine: photonstorm.github.io/phaser3-docs/Phaser.Physics.Arcade.ArcadePhysics.html but it appears to be simpler/less robust (but also possibly faster).

TODO any 2D first person examples a bit like Ciro's 2D reinforcement learning games?

The examples are present under:

git clone https://github.com/photonstorm/phaser3-examples

but note that that repo is huge, about 4.5 GiB on local disk, as is has tons of assets.

The demos also include a Monaco-editor based sandbox mode where you can edit code directly on the web and see the game update which is a really sweet addition.

 Read the full article

Phaser hello world by

Ciro Santilli 40 Updated 2025-07-16

 View more

phaser/hello.html: a minimal hello world adapted from web.archive.org/web/20230323212804/https://phaser.io/tutorials/getting-started-phaser3/part5. Not an actual game strictly speaking though, just shows the phaser logo bouncing around the screen.
Fully contained in a single HTML file, except that it loads the phaser library and image assets from the web.
phaser/hello-game.html: an actually hello world game where you have to collect stars and avoid bombs.
Based on labs.phaser.io/index.html?dir=games/firstgame/&q=:
- finished version: labs.phaser.io/view.html?src=src/games/firstgame/part10.js
- corresponding tutorial: web.archive.org/web/20230323210501/https://phaser.io/tutorials/making-your-first-phaser-3-game/part10.

A web server is mandatory for assets, unless you embed them in data URLs, file:// access is not possible:

 Read the full article

Matter.js by

Ciro Santilli 40 Updated 2025-07-16

 View more

To run the demos locally, tested on Ubuntu 22.10:

git clone https://github.com/liabru/matter-js
cd matter-js
git checkout 0.19.0
npm install
npm run dev

and this opens up the demos on the browser.

 Read the full article

js/matterjs/hello.html by

Ciro Santilli 40 Updated 2025-07-16

 View more

Hello world adapted from: github.com/liabru/matter-js/wiki/Getting-started/1d138998f05766dc4de0e44ae2e35d03121bb7f2

Also asked at: stackoverflow.com/questions/28079138/how-to-make-minimal-example-of-matter-js-work/76203103#76203103

Renderer questions:

follow object on viewport: codepen.io/csims314/pen/goZQvG
draw text: github.com/liabru/matter-js/issues/321

 Read the full article

async (JavaScript) by

Ciro Santilli 40 Updated 2025-07-16

 View more

developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function

async is all present in JavaScript for two reasons:

you make network requests all the time
JavaScript is single threaded, so if you are waiting for a network request, the UI freezes, see remarks on the deprecation of synchronous HTTP request at: developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests

However, it is also Hell: how to convert async to sync in JavaScript.

 Read the full article

Common Crawl by

Ciro Santilli 40 Updated 2025-07-16

 View more

commoncrawl.org/

Amazing project, that basically makes a more searchable Wayback Machine.

A bit hard to use their data though, partly due to size, but also lack of free to use querrying mechanisms, and how obtuse Amazon S3 is to use.

Notably, aws-cli with an account is the only reliable way, everything else is way too broken, e.g. trying the to check the an index index.commoncrawl.org/CC-MAIN-2023-06/ very often 500s.

But still, their projct is amazing.

The only out-of-the-box search they seem to have is: urlsearch.commoncrawl.org/ for domains/URLs. It is good, but there could be so much more... notably IPs.

Also could should document the data shape a bit better.

Sample sizes can be found at: commoncrawl.org/2023/04/mar-apr-2023-crawl-archive-now-available/

To explore the data, after login:

aws s3 ls s3://commoncrawl/crawl-data/CC-MAIN-2013-20/

Copy the toplevel directory only:

aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/ . --recursive --exclude "*/*"

Copy some wet/wat files:

aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz .
aws s3 sync s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz .

Directory structrure:

cc-index.paths.gz (1K)
cc-index-table.paths.gz (1K)

segment.paths.gz (1.7K) Sample lines:

crawl-data/CC-MAIN-2013-20/segments/1368696381249/
crawl-data/CC-MAIN-2013-20/segments/1368696381630/

index.html (2.3K)

wat.paths.gz (98K) Sample lines:

crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wat/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wat.gz

wet.paths.gz (98K) Sample lines:

crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/wet/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.wet.gz

warc.paths.gz (99K)

crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00001-ip-10-60-113-184.ec2.internal.warc.gz

segments: directgory with actual data

1368696381249: one of many segments, any meaning of name?

CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wet.gz (142M, 334M unzipped)

A tiny bit of metadata, and then plaintext content from the website, e.g. the second one:

WARC/1.0
WARC-Type: conversion
WARC-Target-URI: http://004eeb5.netsolhost.com/stephensilver.htm
WARC-Date: 2013-05-18T08:11:02Z
WARC-Record-ID: <urn:uuid:773b31ba-ddc6-47a5-ae24-d08141b9944d>
WARC-Refers-To: <urn:uuid:4b1bdbff-4926-4ced-86f6-072f5bb3837a>
WARC-Block-Digest: sha1:LQFSCR2LIJQYMPTXRHWU7HAPQTVSYS3A
Content-Type: text/plain
Content-Length: 12046

Stephen Silver is a journalist and editor who specializes in the areas of politics, pop culture, film and sports. He works as an editor with the North American Publishing Co. and as a film critic with The Trend, a local newspaper in the Philadelphia area.

No IP unfortunately.

CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.wat.gz (329M, 1.4G unzipped)

A lot of JSON metadata and no contents as desired. Contains IP! Some entries however are humongous with a ton of useless data, that's what bloats these so much:

WARC/1.0
WARC-Type: metadata
WARC-Target-URI: CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz
WARC-Date: 2013-11-22T14:51:12Z
WARC-Record-ID: <urn:uuid:ec54e493-8965-41be-b344-07596cc30b3a>
WARC-Refers-To: <urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>
Content-Type: application/json
Content-Length: 1180

{"Envelope":{"Format":"WARC","WARC-Header-Length":"274","Block-Digest":"sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR","Actual-Content-Length":"372","WARC-Header-Metadata":{"WARC-Type":"warcinfo","WARC-Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz","WARC-Date":"2013-11-22T14:51:12Z","Content-Length":"372","WARC-Record-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Type":"application/warc-fields"},"Payload-Metadata":{"Trailing-Slop-Length":"0","Actual-Content-Type":"application/warc-fields","Actual-Content-Length":"372","Headers-Corrupt":true,"WARC-Info-Metadata":{"robots":"classic","software":"Nutch 1.6 (CC)/CC WarcExport 1.0","description":"Wide crawl of the web with URLs provided by Blekko for Spring 2013","hostname":"ip-10-60-113-184.ec2.internal","format":"WARC File Format 1.0","isPartOf":"CC-MAIN-2013-20","operator":"CommonCrawl Admin","publisher":"CommonCrawl"}}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"453","Header-Length":"10","Inflated-CRC":"866052549","Inflated-Length":"650"},"Offset":"0","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}

WARC/1.0
WARC-Type: metadata
WARC-Target-URI: http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions
WARC-Date: 2013-05-18T05:48:54Z
WARC-Record-ID: <urn:uuid:d519658f-7a63-46c1-849b-4cd92332ddb8>
WARC-Refers-To: <urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>
Content-Type: application/json
Content-Length: 1501

{"Envelope":{"Format":"WARC","WARC-Header-Length":"433","Block-Digest":"sha1:B2B6JDSGWCUQIIUGV54SXEE25RX4SANS","Actual-Content-Length":"302","WARC-Header-Metadata":{"WARC-Type":"request","WARC-Date":"2013-05-18T05:48:54Z","WARC-Warcinfo-ID":"<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>","Content-Length":"302","WARC-Record-ID":"<urn:uuid:cefd363b-1fec-4590-8305-4c6fab2e095f>","WARC-Target-URI":"http://%20jwashington@ap.org/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions","WARC-IP-Address":"165.1.125.44","Content-Type":"application/http; msgtype=request"},"Payload-Metadata":{"Trailing-Slop-Length":"4","HTTP-Request-Metadata":{"Headers":{"Accept-Language":"en-us,en-gb,en;q=0.7,*;q=0.3","Host":"ap.org","Accept-Encoding":"x-gzip, gzip, deflate","User-Agent":"CCBot/2.0","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"},"Headers-Length":"300","Entity-Length":"0","Entity-Trailing-Slop-Bytes":"0","Request-Message":{"Method":"GET","Version":"HTTP/1.0","Path":"/Content/Press-Release/2012/How-AP-reported-in-all-formats-from-tornado-stricken-regions"},"Entity-Digest":"sha1:3I42H3S6NNFQ2MSVX7XZKYAYSCX5QBYJ"},"Actual-Content-Type":"application/http; msgtype=request"}},"Container":{"Compressed":true,"Gzip-Metadata":{"Footer-Length":"8","Deflate-Length":"455","Header-Length":"10","Inflated-CRC":"453539965","Inflated-Length":"739"},"Offset":"453","Filename":"CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"}}

Let's beautify one of them to see it better:


{
  "Envelope": {
    "Format": "WARC",
    "WARC-Header-Length": "274",
    "Block-Digest": "sha1:JCZOI4V3UOTXGIRLFMPLW4J2WPLAKGVR",
    "Actual-Content-Length": "372",
    "WARC-Header-Metadata": {
      "WARC-Type": "warcinfo",
      "WARC-Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz",
      "WARC-Date": "2013-11-22T14:51:12Z",
      "Content-Length": "372",
      "WARC-Record-ID": "<urn:uuid:cfeff436-7c4c-4119-aaa4-ec2ce27ad3e1>",
      "Content-Type": "application/warc-fields"
    },
    "Payload-Metadata": {
      "Trailing-Slop-Length": "0",
      "Actual-Content-Type": "application/warc-fields",
      "Actual-Content-Length": "372",
      "Headers-Corrupt": true,
      "WARC-Info-Metadata": {
        "robots": "classic",
        "software": "Nutch 1.6 (CC)/CC WarcExport 1.0",
        "description": "Wide crawl of the web with URLs provided by Blekko for Spring 2013",
        "hostname": "ip-10-60-113-184.ec2.internal",
        "format": "WARC File Format 1.0",
        "isPartOf": "CC-MAIN-2013-20",
        "operator": "CommonCrawl Admin",
        "publisher": "CommonCrawl"
      }
    }
  },
  "Container": {
    "Compressed": true,
    "Gzip-Metadata": {
      "Footer-Length": "8",
      "Deflate-Length": "453",
      "Header-Length": "10",
      "Inflated-CRC": "866052549",
      "Inflated-Length": "650"
    },
    "Offset": "0",
    "Filename": "CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz"
  }
}

Fuck no IP addresses either. But other entries do have it, why not this one?

The reason these can be huge is the HTML-Metadata section which contain all outlinks! gist.github.com/Smerity/e750f0ef0ab9aa366558#file-bbc-pretty-wat-L34

CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz ()

Obtain:

aws s3 cp s3://commoncrawl/crawl-data/CC-MAIN-2013-20/segments/1368696381249/warc/CC-MAIN-20130516092621-00000-ip-10-60-113-184.ec2.internal.warc.gz .

 Read the full article

How to convert async to sync in JavaScript by

Ciro Santilli 40 Updated 2025-07-16

 View more

God, it's impossible! You just have to convert the entire fucking call stack all the way up to async functions. It could mean refactoring hundreds of functions.

To be fair, there is a logic to this, if you put yourself within the crappiness of the JavaScript threading model. And Python is not that much better with its Global Interpreter Lock.

The problem is that async was introduced relatively late, previously we just had to use infinitely deep callback trees, which was worse:

myAsync().then(ret => myAsync2(ret).then(ret2 => myAsync3(re3)))

compared to the new infinitely more readable:

ret = await myAsync()
ret2 = await myAsync2(ret)
ret3 = await myAsync3(ret3)

But now we are in an endless period of transition between both worlds.

It is also worth mentioning that callbacks are still inescapable if you really want to fan out into a non-linear dependency graph, usually with Promise.all:

await Promise.all([
  myAsync(1).then(ret => myAsync2(ret)),
  myAsync(2).then(ret => myAsync2(ret)),
])

Bibliography:

And then, after many many hours of this work, you might notice that the new code is way, way way slower than before, because making small functions async has a large performance impact: madelinemiller.dev/blog/javascript-promise-overhead/. Real world case with a 4x slowdown: github.com/ourbigbook/ourbigbook/tree/async-slow.

Anyways, since you Googled here, you might as well learn the standard pattern to convert callbacks functions into async functions using a promise: stackoverflow.com/questions/4708787/get-password-from-input-using-node-js/71868483#71868483

Figure 1.
async function Teletubbies meme
. Source. TODO find original source.

 Read the full article

Node.js example by

Ciro Santilli 40 Updated 2025-07-16

 View more

Under nodejs:

 Read the full article

nodejs/count.js by

Ciro Santilli 40 Updated 2025-07-16

 View more

This example counts to finity, sleeping 1 second between each count. Related:

stackoverflow.com/questions/14249506/how-can-i-wait-in-node-js-javascript-l-need-to-pause-for-a-period-of-time

 Read the full article

Node.js standard library by

Ciro Santilli 40 Updated 2025-07-16

 Read the full article

 Pinned article: Introduction to the OurBigBook Project

Welcome to the OurBigBook Project! Our goal is to create the perfect publishing platform for STEM subjects, and get university-level students to write the best free STEM tutorials ever.

Everyone is welcome to create an account and play with the site: ourbigbook.com/go/register. We belive that students themselves can write amazing tutorials, but teachers are welcome too. You can write about anything you want, it doesn't have to be STEM or even educational. Silly test content is very welcome and you won't be penalized in any way. Just keep it legal!

Video 1.

Intro to OurBigBook

. Source.

We have two killer features:

topics: topics group articles by different users with the same title, e.g. here is the topic for the "Fundamental Theorem of Calculus" ourbigbook.com/go/topic/fundamental-theorem-of-calculus
Articles of different users are sorted by upvote within each article page. This feature is a bit like:
- a Wikipedia where each user can have their own version of each article
- a Q&A website like Stack Overflow, where multiple people can give their views on a given topic, and the best ones are sorted by upvote. Except you don't need to wait for someone to ask first, and any topic goes, no matter how narrow or broad
This feature makes it possible for readers to find better explanations of any topic created by other writers. And it allows writers to create an explanation in a place that readers might actually find it.
Figure 1.
Screenshot of the "Derivative" topic page
. View it live at: ourbigbook.com/go/topic/derivative
Video 2.
OurBigBook Web topics demo
. Source.
local editing: you can store all your personal knowledge base content locally in a plaintext markup format that can be edited locally and published either:
- to OurBigBook.com to get awesome multi-user features like topics and likes
- as HTML files to a static website, which you can host yourself for free on many external providers like GitHub Pages, and remain in full control
This way you can be sure that even if OurBigBook.com were to go down one day (which we have no plans to do as it is quite cheap to host!), your content will still be perfectly readable as a static site.
Figure 2.
You can publish local OurBigBook lightweight markup files to either https://OurBigBook.com or as a static website
.
Figure 3.
Visual Studio Code extension installation
.
Figure 4.
Visual Studio Code extension tree navigation
.
Figure 5.
Web editor
. You can also edit articles on the Web editor without installing anything locally.
Video 3.
Edit locally and publish demo
. Source. This shows editing OurBigBook Markup and publishing it using the Visual Studio Code extension.
Video 4.
OurBigBook Visual Studio Code extension editing and navigation demo
. Source.
Internal cross file references done right:
Infinitely deep tables of contents:
Figure 6.
Dynamic article tree with infinitely deep table of contents
.
Live URL: ourbigbook.com/cirosantilli/chordate
Descendant pages can also show up as toplevel e.g.: ourbigbook.com/cirosantilli/chordate-subclade

All our software is open source and hosted at: github.com/ourbigbook/ourbigbook

Further documentation can be found at: docs.ourbigbook.com

Feel free to reach our to us for any help or suggestions: docs.ourbigbook.com/#contact

 Site settings

 Pinned article: Introduction to the OurBigBook Project  Hide

 Pinned article: Introduction to the OurBigBook Project