Example: nodejs/sequelize/raw/tree.js
- Implementation agnostic
- stackoverflow.com/questions/192220/what-is-the-most-efficient-elegant-way-to-parse-a-flat-table-into-a-tree
- stackoverflow.com/questions/5508985/recursive-query-for-adjacency-list-to-preorder-tree-traversal-in-sql DBMS agnostic specifically asking not to modify adjacenty list data structure
- Postgres
- stackoverflow.com/questions/67848017/simple-recursive-sql-query
- stackoverflow.com/questions/28688264/how-to-traverse-a-hierarchical-tree-structure-structure-backwards-using-recursiv
- stackoverflow.com/questions/51822070/how-can-postgres-represent-a-tree-of-row-ids
- depth first
- uspecified depth first variant
- preorder DFS
- breadth-first stackoverflow.com/questions/3709292/select-rows-from-table-using-tree-order
- MySQL
- stackoverflow.com/questions/8252323/mysql-closure-table-hierarchical-database-how-to-pull-information-out-in-the-c asks how to use a specific order (preorder DFS) with closure table
- Microsoft SQL Server
Their crash system does not have an amazing user interface.
Tested on Ubuntu 21.10.
After something crashes, look under
/var/crash for a crash file, which helps to determine which package to report under on Launchpad.E.g. a file
/var/crash/_usr_sbin_gdm3.0.crash makes you want to file the bug under gdm at: bugs.launchpad.net/ubuntu/+source/gdm/+filebugThen, while reporting the bug, you want to give the developpers access to that Ubuntu's crash report system has already uploaded the
.crash file. But you can't publicly upload it because it contains memory dumps and could contain secret information. The way to do it is to look at the ID under:sudo cat /var/crash/_usr_sbin_gdm3.0.uploaded.crash for you, so you just have to confirm it and give the ID on the ticket.You can view a list of all your uploaded errors at:and each of those contain a link to:which you yourself cannot see.
xdg-open https://errors.ubuntu.com/user/$(sudo cat /var/lib/whoopsie/whoopsie-id)https://errors.ubuntu.com/oops/<.uloaded error id>askubuntu.com/questions/434431/how-can-i-read-a-crash-file-from-var-crash asks how to read the
.crash files.Running:splits it up into a few files, but does not make any major improvements.
sudo apport-unpack /var/crash/_usr_sbin_gdm3.0.crash /tmp/appapport-retracesudo apt install apport-retrace
sudo chmod 666 /var/crash/_usr_sbin_gdm3.0.crash
apport-retrace -g /var/crash/_usr_sbin_gdm3.0.crashTried:but then
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
echo -e "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse\ndeb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt install ubuntu-dbgsym-keyringsudo apt update fails with:E: The repository 'http://ddebs.ubuntu.com impish-security Release' does not have a Release file.visualizing the Riemann hypothesis and analytic continuation by 3Blue1Brown (2016) is a good quick visual non-mathematical introduction is to it.
One of the Millennium Prize Problems and Hilbert's problems.
What is the Riemann hypothesis REALLY about? by HexagonVideos (2022)
Source. .data is section 1:00000080 01 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 00 00 00 00 02 00 00 00 00 00 00 |................|
000000a0 0d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|- 80 0:
sh_name=01 00 00 00: index 1 in the.shstrtabstring tableHere,1says the name of this section starts at the first character of that section, and ends at the first NUL character, making up the string.data..datais one of the section names which has a predefined meaning according to www.sco.com/developers/gabi/2003-12-17/ch4.strtab.html:These sections hold initialized data that contribute to the program's memory image.
- 80 4:
sh_type=01 00 00 00:SHT_PROGBITS: the section content is not specified by ELF, only by how the program interprets it. Normal since a.datasection. - 80 8:
sh_flags=037x00:SHF_WRITEandSHF_ALLOC: www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html#sh_flags, as required from a.datasection - 90 0:
sh_addr= 8x00: TODO: standard says:but I don't understand it very well yet.If the section will appear in the memory image of a process, this member gives the address at which the section's first byte should reside. Otherwise, the member contains 0.
- 90 8:
sh_offset=00 02 00 00 00 00 00 00=0x200: number of bytes from the start of the program to the first byte in this section - a0 0:
sh_size=0d 00 00 00 00 00 00 00If we take 0xD bytes starting atsh_offset200, we see:00000200 48 65 6c 6c 6f 20 77 6f 72 6c 64 21 0a 00 |Hello world!.. |AHA! So our"Hello world!"string is in the data section like we told it to be on the NASM.Once we graduate fromhd, we will look this up like:readelf -x .data hello_world.owhich outputs:Hex dump of section '.data': 0x00000000 48656c6c 6f20776f 726c6421 0a Hello world!.NASM sets decent properties for that section because it treats.datamagically: www.nasm.us/doc/nasmdoc7.html#section-7.9.2Also note that this was a bad section choice: a good C compiler would put the string in.rodatainstead, because it is read-only and it would allow for further OS optimizations.- a0 8:
sh_linkandsh_info= 8x 0: do not apply to this section type. www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html#special_sections - b0 0:
sh_addralign=04= TODO: why is this alignment necessary? Is it only forsh_addr, or also for symbols insidesh_addr? - b0 8:
sh_entsize=00= the section does not contain a table. If != 0, it means that the section contains a table of fixed size entries. In this file, we see from thereadelfoutput that this is the case for the.symtaband.rela.textsections.
- a0 8:
The minimalism, serverlessness/lack of temporary caches/lack of permission management, Hipp's religious obsession with efficiency, the use of their own pure Fossil version control[ref]. Wait, scrap that last one. Pure beauty!
Official Git mirror: github.com/sqlite/sqlite
Create a table
sqlite3 db.sqlite3 "
CREATE TABLE 'IntegerNames' (int0 INT, char0 CHAR(16));
INSERT INTO 'IntegerNames' (int0, char0) VALUES (2, 'two'), (3, 'three'), (5, 'five'), (7, 'seven');
"List tables:output:
sqlite3 db.sqlite3 '.tables'IntegerNamesShow schema of a table:outputs the query that would generate that table:
sqlite3 db.sqlite3 '.schema IntegerNames'CREATE TABLE IF NOT EXISTS 'IntegerNames' (int0 INT, char0 CHAR(16));Show all data in a table:output:
sqlite3 db.sqlite3 'SELECT * FROM IntegerNames'2|two
3|three
5|five
7|sevenpytorch.org/vision/0.13/models.html has a minimal runnable example adapted to python/pytorch/resnet_demo.py.
That example uses a ResNet pre-trained on the COCO dataset to do some inference, tested on Ubuntu 22.10:This first downloads the model, which is currently 167 MB.
cd python/pytorch
wget -O resnet_demo_in.jpg https://upload.wikimedia.org/wikipedia/commons/thumb/6/60/Rooster_portrait2.jpg/400px-Rooster_portrait2.jpg
./resnet_demo.py resnet_demo_in.jpg resnet_demo_out.jpgWe know it is COCO because of the docs: pytorch.org/vision/0.13/models/generated/torchvision.models.detection.fasterrcnn_resnet50_fpn_v2.html which explains that is an alias for:
FasterRCNN_ResNet50_FPN_V2_Weights.DEFAULTFasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1The runtime is relatively slow on P51, about 4.7s.
After it finishes, the program prints the recognized classes:so we get the expected
['bird', 'banana']bird, but also the more intriguing banana.By looking at the output image with bounding boxes, we understand where the banana came from!
python/pytorch/resnet_demo_out.jpg
. The beak was of course a banana, not a beak!Paging makes it easier to compile and run two programs or threads at the same time on a single computer.
For example, when you compile two programs, the compiler does not know if they are going to be running at the same time or not.
So nothing prevents it from using the same RAM address, say,
0x1234, to store a global variable.And thread stacks, that must be contiguous and keep growing down until they overwrite each other, are an even bigger issue!
But if two programs use the same address and run at the same time, this is obviously going to break them!
Paging solves this problem beautifully by adding one degree of indirection:
(logical) ------------> (physical)
pagingWhere:
- logical addresses are what userland programs see, e.g. the contents of
rsiinmov eax, [rsi].They are often called "virtual" addresses as well. - physical addresses can be though of the values that go to physical RAM index wires.But keep in mind that this is not 100% true because of further indirections such as:
Compilers don't need to worry about other programs: they just use simple logical addresses.
As far as programs are concerned, they think they can use any address between 0 and 4GiB (2^32,
FFFFFFFF) on 32-bit systems.The OS then sets up paging so that identical logical addresses will go into different physical addresses and not overwrite each other.
This makes it much simpler to compile programs and run them at the same time.
Paging achieves that goal, and in addition:
- the switch between programs is very fast, because it is implemented by hardware
- the memory of both programs can grow and shrink as needed without too much fragmentation
- one program can never access the memory of another program, even if it wanted to.This is good both for security, and to prevent bugs in one program from crashing other programs.
Or if you like non-funny jokes:
Comparison between the Linux kernel userland memory virtualization and The Matrix
. Source. Is this RAM real?
Ciro Santilli likes to learn astronomy a bit like he learns geography: go down some lists of "stuff that seems most relevant in some criteria to us!", possibly at different size scales e.g.:
Implosion-type fission weapons are more complicated than gun-type fission weapon because you have to precisely coordinate the detonation of a bunch of explosives.
Subset generators:
- github.com/mf1024/ImageNet-datasets-downloader generates on download, very good. As per github.com/mf1024/ImageNet-Datasets-Downloader/issues/14 counts go over the limit due to bad multithreading. Also unfortunately it does not start with a subset of 1k.
- github.com/BenediktAlkin/ImageNetSubsetGenerator
Unfortunately, since ImageNet is a closed standard no one can upload such pre-made subsets, forcing everybody to download the full dataset, in ImageNet1k, which is huge!
Array of
Elf64_Shdr structs.Each entry contains metadata about a given section.
e_shoff of the ELF header gives the starting position, 0x40 here.e_shentsize and e_shnum from the ELF header say that we have 7 entries, each 0x40 bytes long.So the table takes bytes from 0x40 to
0x40 + 7 + 0x40 - 1 = 0x1FF.Some section names are reserved for certain section types: www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html#special_sections e.g.
.text requires a SHT_PROGBITS type and SHF_ALLOC + SHF_EXECINSTRRunning:outputs:
readelf -S hello_world.oThere are 7 section headers, starting at offset 0x40:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .data PROGBITS 0000000000000000 00000200
000000000000000d 0000000000000000 WA 0 0 4
[ 2] .text PROGBITS 0000000000000000 00000210
0000000000000027 0000000000000000 AX 0 0 16
[ 3] .shstrtab STRTAB 0000000000000000 00000240
0000000000000032 0000000000000000 0 0 1
[ 4] .symtab SYMTAB 0000000000000000 00000280
00000000000000a8 0000000000000018 5 6 4
[ 5] .strtab STRTAB 0000000000000000 00000330
0000000000000034 0000000000000000 0 0 1
[ 6] .rela.text RELA 0000000000000000 00000370
0000000000000018 0000000000000018 4 2 4
Key to Flags:
W (write), A (alloc), X (execute), M (merge), S (strings), l (large)
I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)
O (extra OS processing required) o (OS specific), p (processor specific)The
struct represented by each entry is:typedef struct {
Elf64_Word sh_name;
Elf64_Word sh_type;
Elf64_Xword sh_flags;
Elf64_Addr sh_addr;
Elf64_Off sh_offset;
Elf64_Xword sh_size;
Elf64_Word sh_link;
Elf64_Word sh_info;
Elf64_Xword sh_addralign;
Elf64_Xword sh_entsize;
} Elf64_Shdr;GDM crashes sometimes when switching windows right after opening a new window: bugs.launchpad.net/ubuntu/+source/gdm/+bug/1956299
Paging is implemented by the CPU hardware itself.
Paging could be implemented in software, but that would be too slow, because every single RAM memory access uses it!
Operating systems must setup and control paging by communicating to the CPU hardware. This is done mostly via:
- the CR3 register, which tells the CPU where the page table is in RAM memory
- writing the correct paging data structures to the RAM pointed to the CR3 register.Using RAM data structures is a common technique when lots of data must be transmitted to the CPU as it would cost too much to have such a large CPU register.The format of the configuration data structures is fixed _by the hardware_, but it is up to the OS to set up and manage those data structures on RAM correctly, and to tell the hardware where to find them (via
cr3).Then some heavy caching is done to ensure that the RAM access will be fast, in particular using the TLB.Another notable example of RAM data structure used by the CPU is the IDT which sets up interrupt handlers.The OS makes it impossible for programs to change the paging setup directly without going through the OS: - CR3 cannot be modified in ring 3. The OS runs in ring 0. See also:
- the page table structures are made invisible to the process using paging itself!
Processes can however make requests to the OS that cause the page tables to be modified, notably:
- stack size changes
brkandmmapcalls, see also: stackoverflow.com/questions/6988487/what-does-brk-system-call-do/31082353#31082353
The kernel then decides if the request will be granted or not in a controlled manner.
This was Ciro Santilli's main study/work music for several years around 2020. Tabla rules.
Unlisted articles are being shown, click here to show only listed articles.