Funding:
- 2023: 1.1m pounds www.uktech.news/deep-tech/lumai-grant-20230215
The highly underdocumented built-in module, that supports SQL spatial index and a lot more.
Quite horrendous as it only seems to work on geometric types and not existing columns. But why.
And it uses custom operatores, where standard operators would have been just fine for points...
Minimal runnable example with points:The index creation unfortunately took 100s, so it will not scale to 1B points very well whic his a shame.
set -x
time psql -c 'drop table if exists t'
time psql -c 'create table t(p point)'
time psql -c "insert into t select (point ('(' || generate_series || ',' || generate_series || ')')) from generate_series(1, 10000000)"
time psql -c 'create index on t using gist(p)'
time psql -c "select count(*) from t where p <@ box '(1000000,1000000),(9000000,2000000)'"The third part module, which clutters up any serches you make for the built-in one.
@cirosantilli/_file/python/python/getitem.py by 
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01 @cirosantilli/_file/python/python/getitem_complex.py by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01Example: nodejs/sequelize/trigger_count.js
There is of course no built-in support for SQL TRIGGERs in Sequelize, but we can add our own: stackoverflow.com/questions/29716346/how-to-create-a-trigger-in-sequelize-nodejs/76215728#76215728
- From Motor Control to Team Play in Simulated Humanoid Football
From Motor Control to Team Play in Simulated Humanoid Football by Ali Eslami (2023)
Source. Likely a reupload by DeepMind employee: www.linkedin.com/in/smalieslami.DeepMind’s AI Trained For 5 Years by Two Minute Papers (2023)
Source. The 5 years bullshit is of course in-game time clickbait, they simulate 1000x faster than realtime. Ubuntu 23.04 boot broken on kernel 6.2 by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01Switching to the other installed kernel, 5.9 made boot work.
The solution on kernel 6.2 was:as per comments under: bugs.launchpad.net/ubuntu/+source/linux/+bug/2012559. This also made the nvidia driver work: Find GPU information in Ubuntu.
sudo apt instal nvidia-driver-515Previously I had:and it blew up before reaching disk decryption.
nvidia-driver-510I also tried:but that broke in a different way:
nvidia-driver-525Finished apport-autoreport.service - Process error reports when automatic reporting is enabled.
nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000947d:0:0:407
(1 of 2) Job systemd-backlight@backlight: nvidia_e.service/start running (32s no limit)Demo under: nodejs/sequelize/raw/many_to_many.js.
NO way in the SQL standard apparently, but you'd hope that implementation status would be similar to UPDATE with JOIN, but not even!
- PostgreSQL: possible with
DELETE FROM USING: stackoverflow.com/questions/11753904/postgresql-delete-with-inner-join - SQLite: not possible without subqueries as of 3.35 far: stackoverflow.com/questions/24511153/how-delete-table-inner-join-with-other-table-in-sqlite, Does not appear to have any relevant features at: www.sqlite.org/lang_delete.html
ORM
- Sequelize: no support of course: stackoverflow.com/questions/40890131/sequelize-destroy-record-with-join
Attempt at nodejs/sequelize/raw/upsert.js:
- stackoverflow.com/questions/48816629/on-conflict-do-nothing-in-postgres-with-a-not-null-constraint OP unable to provide a minimal exampe, but it is likely the problem
- dba.stackexchange.com/questions/292428/postgresql-upsert-issue-with-not-null-columns
AlphaGo Zero open source implementation by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01TODO vs all the others?
Quantum approximate optimization algorithm by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01TODO clear example of the computational problem that it solves.
Quadratic unconstrained binary optimization by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01Exampes under nodejs/sequelize/raw:
- nodejs/sequelize/raw/index.js: Sequelize raw query hello world. Ideally one should never use a raw query in a real project. We use raw examples mostly as a SQL tutorial under SQL example, and will not comment on them much further on this section.
@cirosantilli/_file/nodejs/sequelize/nodejs/sequelize/parallel_select_and_update.js by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01This example is the same as nodejs/sequelize/raw/parallel_select_and_update.js, but going through Sequelize rather than with Sequelize raw queries.
NONE is not supported for now to not have a transaction at all because lazy.The examples illustrates: stackoverflow.com/questions/55452441/for-share-and-for-update-statements-in-sequelize
Sample invocation:where:
node --unhandled-rejections=strict ./parallel_select_and_update.js p 10 100 READ_COMMITTED UPDATEREAD_COMMITTED: one of the keys documented at: sequelize.org/master/class/lib/transaction.js~Transaction.html which correspond to the standard sQL isolation levels. It not given, don't set one, defaulting to the database's/sequelize's default level.UPDATE: one of the keys documented at: sequelize.org/master/class/lib/transaction.js~Transaction.html#static-get-LOCK. Update generates aSELECT FOR UPDATEin PostgreSQL for example. If not given, don't use anyFOR xxxexplicit locking.
Other examples:
node --unhandled-rejections=strict ./parallel_select_and_update.js p 10 100 READ_COMMITTED UPDATE
Then, the outcome is exactly as described at: nodejs/sequelize/raw/parallel_select_and_update.js:
READ_COMMITTED: failsREAD_COMMITTED UPDATE: worksREPEATABLE_READ: works, but is a bit slower, as it does rollbacksThis case also illustrates Sequelize transaction retries, since in this transaction isolation level transactions may fail:
@cirosantilli/_file/nodejs/sequelize/raw/nodejs/sequelize/raw/parallel_update_async.js by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01nodejs/sequelize/raw/parallel_update_worker_threads.js contains a base example that can be used to test what can happen when queries are being run in parallel. But it is broken due to a
sqlite3 Node.js package bug: github.com/mapbox/node-sqlite3/issues/1381...nodejs/sequelize/raw/parallel_update_async.js is an
async version of it. It should be just parallel enough to allow observing the same effects.This is an example of a transaction where the SQL READ COMMITTED isolation level if sufficient.
These examples run queries of type:
UPDATE "MyInt" SET i = i + 1Sample execution:which does:
node --unhandled-rejections=strict ./parallel_update_async.js p 10 100- PostgreSQL, see other databases options at SQL example
- 10 threads
- 100 increments on each thread
The fear then is that of a classic read-modify-write failure.
But as www.postgresql.org/docs/14/transaction-iso.html page makes very clear, including with an explicit example of type
UPDATE accounts SET balance = balance + 100.00 WHERE acctnum = 12345;, that the default isolation level, SQL READ COMMITTED isolation level, already prevents any problems with this, as the update always re-reads selected rows in case they were previously modified.If the first updater commits, the second updater will ignore the row if the first updater deleted it, otherwise it will attempt to apply its operation to the updated version of the row
Since in PostgreSQL "Read uncommitted" appears to be effectively the same as "Read committed", we won't be able to observe any failures on that database system for this example.
nodejs/sequelize/raw/parallel_create_delete_empty_tag.js contains an example where things can actually blow up in read committed.
@cirosantilli/_file/nodejs/sequelize/raw/nodejs/sequelize/raw/parallel_select_and_update.js by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01This example is similar to nodejs/sequelize/raw/parallel_update_async.js, but now we are doing a separate SELECT, later followed by an update:
SELECT FROMto get i- update on Js code
newI = i + 1 UPDATE SETthenewI
Although this specific example is useless in itself, as we could just use
UPDATE "MyInt" SET i = i + 1 as in nodejs/sequelize/raw/parallel_update_async.js, which automatically solves any concurrency issue, this kind of code could be required for example if the update was a complex function not suitably implemented in SQL, or if the update depends on some external data source.Sample execution:which does:
node --unhandled-rejections=strict ./parallel_select_and_update.js p 2 10 'READ COMMITTED'- PostgreSQL, see other databases options at SQL example
- 2 threads
- 10 increments on each thread
Another one:this will run SELECT FOR UPDATE rather than just SELECT
node --unhandled-rejections=strict ./parallel_select_and_update.js p 2 10 'READ COMMITTED' 'FOR UPDATE'Observed behaviour under different SQL transaction isolation levels:
READ COMMITTED: fails. Nothing in this case prevents:- thread 1: SELECT, obtains i = 0
- thread 2: SELECT, obtains i = 0
- thread 2: newI = 1
- thread 2: UPDATE i = 1
- thread 1: newI = 1
- thread 1: UPDATE i = 1
REPEATABLE READ: works. the manual mentions that if multiple concurrent updates would happen, only the first commit succeeds, and the following ones fail and rollback and retry, therefore preventing the loss of an update.READ COMMITTED+SELECT FOR UPDATE: works. And does not do rollbacks, which probably makes it faster. Withp 10 100,REPEATABLE READwas about 4.2s andREAD COMMITTED+SELECT FOR UPDATE3.2s on Lenovo ThinkPad P51 (2017).SELECT FOR UPDATEshould be enough as mentioned at: www.postgresql.org/docs/13/explicit-locking.html#LOCKING-ROWSFOR UPDATE causes the rows retrieved by the SELECT statement to be locked as though for update. This prevents them from being locked, modified or deleted by other transactions until the current transaction ends. That is, other transactions that attempt UPDATE, DELETE, SELECT FOR UPDATE, SELECT FOR NO KEY UPDATE, SELECT FOR SHARE or SELECT FOR KEY SHARE of these rows will be blocked until the current transaction ends; conversely, SELECT FOR UPDATE will wait for a concurrent transaction that has run any of those commands on the same row, and will then lock and return the updated row (or no row, if the row was deleted). Within a REPEATABLE READ or SERIALIZABLE transaction, however, an error will be thrown if a row to be locked has changed since the transaction started. For further discussion see Section 13.4.
A non-raw version of this example can be seen at: nodejs/sequelize/parallel_select_and_update.js.
@cirosantilli/_file/nodejs/sequelize/raw/nodejs/sequelize/raw/parallel_select_and_update_deterministic.js by Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01
Ciro Santilli 35 Updated 2024-12-23 +Created 1970-01-01This example contains a deterministic demo of when postgreSQL serialization failures may happen.
Tested on PostgreSQL 13.5.
There are unlisted articles, also show them or only show them.