Ciro's word of caution for 2019 aspiring system programmers: Should you waste your life with systems programming?
This is basically a direct consequence of backward design.
The higher the level you can operate at, the better.
The ideal level to operate at, and one of humankind's greatest ambitions is "AGI, make me money", the highest possible level.
Only go down a level when it seems necessary.
Of course, "Ciro Santilli" with quotes, since all of those are either taken directly from others, or had been previously formulated by others.
Ciro once read that there are two types of mathematicians/scientists (he thinks it was comparing Einstein to some Jack of all trades polymath who didn't do any new discoveries):
- high flying birds, who know a bit of everything, feel the beauty of each field, but never dig deep in any of them
- gophers, who dig all the way down, on a single subject, until they either get the Nobel Prize, or work on the wrong problem and waste their lives
TODO long after Ciro forgot where he had read this from originally, someone later pointed him to: www.ams.org/notices/200902/rtx090200212p.pdf Birds and Frogs by Freeman Dyson (2009), which is analogous but about Birds and Frogs. So did Ciro's memory play a trick on him, or is there also a variant; of this metaphor with a gopher?
Ciro is without a doubt the bird type. Perhaps the ultimate scientist is the one who can combine both aspects in the right amount?
Ciro gets bored of things very quickly.
Once he understands the general principles, if the thing is not the next big thing, Ciro considers himself satisfied without all the nitty gritty detail, and moves on to the next attempt.
In the field of mathematics for example, Ciro is generally content with understanding cool theorem statements. More generally, one of Ciro's desires is for example to understand the significance of each physics Nobel Prize.
This is also very clear for example after Ciro achieved Linux Kernel Module Cheat: he now had the perfect setup to learn all the Linux kernel shady details but at the same time after all those years he finally felt that "he could do it, so that was enough", and soon moved to other projects.
If Ciro had become a scientist, he would write the best review papers ever, just like in the current reality he writes amazing programming tutorials on Stack Overflow.
Ciro has in his mind an overly large list of subjects that "he feels he should know the basics of", and whenever he finds something in one of those topics that he does not know enough about, he uncontrollably learns it, even if it is not the most urgent thing to be done. Or at least he puts a mention on his "list of sources" about the subject. Maybe everyone is like that. But Ciro feels that he feels this urge particularly strongly. Correspondingly, if a subject is not in that list, Ciro ignores it without thinking twice.
Ciro believes that high flying birds are the type of people better suited for venture capital investment management: you know a bit of what is hot on several fields to enough depth to decide where to place your bets and how to guide them. But you don't have the patience to actually go deeply into any one of them and deal with each individual shit that comes up.
Cosmos: A Personal Voyage (1980) episode 1 mentions as quoted by the Wikipedia page for Eratosthenes:That's Ciro.
According to an entry in the Suda (a 10th-century encyclopedia), his critics scorned him, calling him beta (the second letter of the Greek alphabet) because he always came in second in all his endeavours.
A computer is a highly layered system, and so you have to decide which layers you are the most interested in studying.
Although the layer are somewhat independent, they also sometimes interact, and when that happens it usually hurts your brain. E.g., if compilers were perfect, no one optimizing software would have to know anything about microarchitecture. But if you want to go hardcore enough, you might have to learn some lower layer.
It must also be said that like in any industry, certain layers are hidden in commercial secrecy mysteries making it harder to actually learn them. In computing, the lower level you go, the more closed source things tend to become.
But as you climb down into the abyss of low level hardcoreness, don't forget that making usefulness is more important than being hardcore: Figure 1. "xkcd 378: Real Programmers".
First, the most important thing you should know about this subject: cirosantilli.com/linux-kernel-module-cheat/should-you-waste-your-life-with-systems-programming
Here's a summary from low-level to high-level:
- semiconductor physical implementation this level is of course the most closed, but it is fun to try and peek into it from any openings given by commercials and academia:
- photolithography, and notably photomask design
- register transfer level
- interactive Verilator fun: Is it possible to do interactive user input and output simulation in VHDL or Verilog?
- more importantly, and much harder/maybe impossible with open source, would be to try and set up a open source standard cell library and supporting software to obtain power, performance and area estimates
- Are there good open source standard cell libraries to learn IC synthesis with EDA tools? on Quora
- the most open source ones are some initiatives targeting FPGAs, e.g. symbiflow.github.io/, www.clifford.at/icestorm/
- qflow is an initiative targeting actual integrated circuits
- microarchitecture: a good way to play with this is to try and run some minimal userland examples on gem5 userland simulation with logging, e.g. see on the Linux Kernel Module Cheat:This should be done at the same time as books/website/courses that explain the microarchitecture basics.This is the level of abstraction that Ciro Santilli finds the most interesting of the hardware stack. Learning it for actual CPUs (which as of 2020 is only partially documented by vendors) could actually be useful in hardcore software optimization use cases.
- instruction set architecture: a good approach to learn this is to manually write some userland assembly with assertions as done in the Linux Kernel Module Cheat e.g. at:
- github.com/cirosantilli/linux-kernel-module-cheat/blob/9b6552ab6c66cb14d531eff903c4e78f3561e9ca/userland/arch/x86_64/add.S
- cirosantilli.com/linux-kernel-module-cheat/x86-userland-assembly
- learn a bit about calling conventions, e.g. by calling C standard library functions from assembly:
- you can also try and understand what some simple C programs compile to. Things can get a bit hard though when
-O3is used. Some cute examples:
- executable file format, notably executable and Linkable Format. Particularly important is to understand the basics of:
- address relocation: How do linkers and address relocation work?
- position independent code: What is the -fPIE option for position-independent executables in GCC and ld?
- how to observe which symbols are present in object files, e.g.:
- how C++ uses name mangling What is the effect of extern "C" in C++?
- how C++ template instantiation can help reduce link time and size: Explicit template instantiation - when is it used?
- operating system. There are two ways to approach this:
- learn about the Linux kernel Linux kernel. A good starting point is to learn about its main interfaces. This is well shown at Linux Kernel Module Cheat:
- system calls
- write some system calls in
- pure assembly:
- C GCC inline assembly:
- write some system calls in
- learn about kernel modules and their interfaces. Notably, learn about to demystify special files such
/dev/randomand so on: - learn how to do a minimal Linux kernel disk image/boot to userland hello world: What is the smallest possible Linux implementation?
- learn how to GDB Step debug the Linux kernel itself. Once you know this, you will feel that "given enough patience, I could understand anything that I wanted about the kernel", and you can then proceed to not learn almost anything about it and carry on with your life
- system calls
- write your own (mini-) OS, or study a minimal educational OS, e.g. as in:
- learn about the Linux kernel Linux kernel. A good starting point is to learn about its main interfaces. This is well shown at Linux Kernel Module Cheat:
- programming language
How low can you go video by Ciro Santilli (2017)
Source. In this infamous video Ciro has summarized the computer hierarchy.The approach many courses take to physics, specially "modern Physics" is really bad, this is how it should be taught:
- start by describing experiments that the previous best theory did not explain, see also: Section "Physics education needs more focus on understanding experiments and their history"
- then, give the final formula for the next best theory
- then, give all the important final implications of that formula, and how it amazingly describes the experiments. In particular this means: doing physics means calculating a number
- then, give some mathematical intuition on the formulas, and how the main equation could have been derived
- finally, then and only then, start deriving the outcomes of the main formula in detail
This is likely because at some point, experiments get more and more complicated, and so people are tempted to say "this is the truth" instead of "this is why we think this is the truth", which is much harder.
But we can't be lazy, there is no replacement to the why.
Related:
- settheory.net/learnphysics and www.youtube.com/watch?v=5MKjPYuD60I&list=PLJcTRymdlUQPwx8qU4ln83huPx-6Y3XxH from settheory.net
- math.ucr.edu/home/baez/books.html by John Baez. Mentions:
This webpage doesn't have lots of links to websites. Websites just don't have the sort of in-depth material you need to learn technical subjects like advanced math and physics — at least, not yet. To learn this stuff, you need to read lots of booksCiro Santilli is trying to change that: OurBigBook.com.
- web.archive.org/web/20210324182549/http://jakobschwichtenberg.com/one-thing/ by Jakob Schwichtenberg
In the 2010's/2020's, many people got excited about getting children in to electronics with cheap devboards, notably with Raspberry Pi and Arduino.
While there is some potential in that, Ciro Santilli always felt that this is very difficult to do, while also keeping his sacred principle of backward design in mind.
The reason for this is that "everyone" already has much more powerful computers at hand: their laptops/desktops and even mobile phones as of the 2020s. Except perhaps if you are thing specifically about poor countries.
Therefore, the advantage using such devboards for doing something that could useful must come from either:
- their low cost. This would be an important consideration if you were to mass produce your product, but that is not going to be the case for learners, at least initially.
- their portability, and closely linked their ability to act as sensors
- their ability to act as actuators, which is often missing from regular computers
- them having hardware accelerators that are not normally present in regular computers, e.g. FPGAs or AI accelerators. And then the demo project must demonstrate that the project is able to do something significantly faster/cheaper on the devboard than on a desktop computer.
It boggles Ciro Santilli's mind that people use mailing list to collaborate on projects!
The only explanation is that the dinosaurs who created the projects are unable to adapt to new superior technologies.
Yes, Ciro is talking to you, big fundamental projects from last century: Linux kernel, GNU Compiler Collection (gcc.gnu.org/lists.html), Binutils (sourceware.org/binutils/), etc.
Some of you are already using Bugzilla for the bugs, so kudos. But if you've seen their benefit, why you still use the mailing list for patches?
Advantages of mailing lists:
- threaded replies, which almost no issue tracker has. GitHub feature request: github.com/isaacs/github/issues/837
Disadvantages: everything else:
- cannot subscribed to a single thread. Which forces you to create an email filter for each one of them you subscribe to.
- no metadata, notably the notion of closing / merging, but also upvotesYou have to read thirty messages before you can know if the bug was solved or not.
- it is insanely hard to reply to messages from before you were subscribed: webapps.stackexchange.com/questions/23197/reply-to-mailman-archived-message/115088#115088This forces everyone to subscribe to all lists, and then set up email filters to not be flooded with emails.
- hard to apply patches locally to test them out: stackoverflow.com/questions/5062389/how-to-use-git-am-to-apply-patches-from-email-messages/49082916#49082916Unless they use Patchwork, which adds one more website on top of the mess.And then Gmail corrupts your patches, and you are forced to use
git send-email, which does not work on some network configurations: stackoverflow.com/questions/28038662/how-to-solve-unable-to-initialize-smtp-properly-when-using-using-git-send-ema or setup ThunderBird. - often have to subscribe to post at all, thus cluttering your inbox further
- you can edit posts to make them clearer.Yes, people could vandalize their answers when they get mad, and threads might stop making sense after edits. But this can be solved with an undeletable post history like Stack Overflow has (but not any other tracker does).Or archive.org :-)In any case, what do you think will happen more often and have greater impact:
- people vandalize their posts
- people fix their silly typos and improve content
- searchable by author, keyword, etc. without Google. Yes, mailing list trackers could have decent implementations to overcome that. But no, GNU Mailman which everyone uses does not have it. Google barely indexes it.And I don't think Google properly indexes many of the mailing list archives for some reason: I never get hits for my own posts a week later, while I often do on GitHub issues.
- people have to learn about top posting vs inline posting, and this requires infinite education of new users
- Line comments in code reviews like GitHub and GitLab.On mailing lists: either put a comment in the middle of a huge patch and let other people find it, or (more likely) copy paste the part of the patch that you are talking about.
- most mail web UIs suck.OK, this is not an unsolvable or intrinsic problem, but still a problem.E.g.:
ezmlmit is not possible to see the entire content in a single page: gcc.gnu.org/ml/gcc/2015-07/threads.html.Unless you like reading threads backwards and with 4 levels of>quotations.The alternative: do like LLVM and send attachments. Yes, I we all love opening up attachments on our browsers.The real solution: everyone can create branches and pull requests. Also has the benefit of running CI on the pull requests.
Not sure:
- you can have infinitely many trackers to replicate data in case apocalypse happens in some part of the world.Although I'm not sure this is an advantage, as you don't know anymore which one is the canonical trackers an advantage, as you don't know anymore which one is the canonical tracker.And all web interfaces already have an API to export messages, and someone has already scripted it to import from any web UI to any web UI for you.And GitHub offers infinite precise history transparently on its API.
The natural sciences are not just a tool to predict the future.
They are a reminder that the lives that we live daily are mere illusions, religious concepts such as Maya and Samsara come to mind.
We as individuals perceive nothing about the materials that we touch every day really work, nor more importantly how our brain and cell work.
Everything is magic out of our control.
The natural sciences allow us peek, with huge concentrated effort, into tiny little bits a little of those unknowns, and blow our minds as we notice that we don't know anything.
For all practical purposes in life, there is a huge macro micro gap. We are only able to directly perceive and influence the macro events. And through those we try to affect micro events. Because for good or bad, micro events reflect in the macro world.
It is as if we live in a different plane of existence above molecules, and below galaxies. The hierarchy of Figure "xkcd 435: Fields arranged by purity" puts that nicely into perspective, shame it only starts at the economical level, not going up to astronomy.
The great beauty of science is that it allows us to puncture through some of the layers of reality, either up or down, away from our daily experience.
And the great beauty of artificial intelligence research is that it allows to peer deeper into exactly our layer of existence.
Every one or two weeks Ciro Santilli remembers that he and everything he touches are just a bunch of atoms, and that is an amazing feeling. This is Ciro's preferred source of Great doubt. Another concept that comes to mind is when you see it, you'll shit bricks.
Perhaps, the feeling of physics and the illusion of life reaches its peak in molecular biology.
Just look at your fucking hand right now.
Do you have any idea of each of the cells in it work? Isn't is at least 100 times more complex than the materials of the table you hand is currently resting on?
This is the non-science fiction version of the lotus-Eater Machine.
Alan Watts's "Philosopher" talk mentions related ideas:
The origin of a person who is defined as a philosopher, is one who finds that existence itself is exceedingly odd.
The toddler of a friend of Ciro Santilli's wife asked her mum:Our perception of the macroscopic world is so magic that children have to learn the difference between living and non-living things.
Why doesn't my tiger doll close its eyes when we sleep?
James Somers put it very well as well in his article I should have loved biology by James Somers, this quote was brought to Ciro's attention by Bert Hubert's website[ref].The same applies to other natural sciences.
I should have loved biology but I found it to be a lifeless recitation of names: the Golgi apparatus and the Krebs cycle; mitosis, meiosis; DNA, RNA, mRNA, tRNA.In the textbooks, astonishing facts were presented without astonishment. Someone probably told me that every cell in my body has the same DNA. But no one shook me by the shoulders, saying how crazy that was. I needed Lewis Thomas, who wrote in The Medusa and the Snail:For the real amazement, if you wish to be amazed, is this process. You start out as a single cell derived from the coupling of a sperm and an egg; this divides in two, then four, then eight, and so on, and at a certain stage there emerges a single cell which has as all its progeny the human brain. The mere existence of such a cell should be one of the great astonishments of the earth. People ought to be walking around all day, all through their waking hours calling to each other in endless wonderment, talking of nothing except that cell.
Alan Watts' "Philosopher" talk (1973)
Source. Lecture given at UCLA on 1973-02-21. Some key quotes from the talk:The origin of a person who is defined as a philosopher, is one who finds that existence itself is exceedingly odd.
Once upon a time in the 2010's, Ciro Santilli went to an artsy theatre venue in the suburbia of Paris, dragged by his wife then girlfriend of course.
In the venue, there was a politician, who was doing his best to show how much they supported the arts, and there were of course the artists, involved in the play.
The politician would see a political power score on top of every person's head, and would spend an amount of time talking to each person exactly proportional to that score. This meant basically one sentence to us. The words themselves didn't really matter of course, only the time spent, they just have to produce nice sounds.
One of the artists however, and he seemed quite important in the production, for some reason spent a huge amount of time speaking to us. The score the artist saw on our heads was of love, or how interested we were in the art.
Big goals:
- the pursuit of AGI
- physics simulations, including scientific visualization software
- formalization of mathematics
Ciro Santilli believes that there is a positive correlation between being a software engineer and liking Buddhist-like things.
Maybe it is linked to minimalism and DRY, which software engineers value so greatly.
Even Ciro had to try an unoriginal Buddhist joke intro in one of this Stack Overflow answers.
Ciro also feels that his "minimal reproducible example" scientific language/concept learning method obsession of breaking things into tiny sub-problems has a strong link with Koans.
Some notable Buddhism/programmer examples:
- www.catb.org/~esr/writings/unix-koans/ "The Unix Koans of Master Foo - Rootless Root (无根的根)" by the legendary Eric Steven Raymond is notable
- thecodelesscode.com/ "The Codeless Code" by anonymous Qi.
- canonical.org/~kragen/tao-of-programming.html
- wiki.c2.com/?MysticalProgrammingKoans
- rubykoans.com/ even evil programming languages adopt them!
- The Zen of Python
Another thing that points the correlation out is the existence of wattsalan.github.io/ on a
github.io about Alan Watts.Ciro once commented that the best game is an infinitely hard one, where you can progress infinitely. To which his great friend J. replied:Or more broadly, one may argue that the perfect video game is life itself, or difficult life goals like making money, becoming famous or changing the world.
Fine, so the perfect game for you is mathematics. Stage one: prove the Riemann hypothesis!
Thinking about it, "infinitely hard" is perhaps not a very precise term, as it could be interpreted as impossible. And if you have mathematical proof that something is impossible, it would be "pointless" to try, trying would be equivalent to pure meditation.
Maybe a better way to put it would be in terms of a difficulty curve. Real life also involves a lot of waiting, either for some experiment to finish running, of for you mental energy to restore a bit.
But so be it, you get the idea.
But this is basically what Ciro feels on every video game. It happens too often on PVE games that things are is either:
- too slow and easy (Ciro would rather skip those with saves made by other)
- or too fast hard, Ciro would rather tool-assisted speedrun those parts
Not to mention the incredible breach of suspension of disbelief of most PvE games where enemies are unbelievably stupid. E.g., why doesn't Bowser just build one fucking wall 15 tiles high to prevent Mario from coming through to his castle? And then put a gate and a hundred guards in front of it? TODO there was a YouTube video of this, I think it was Toad pointing it out to Mario that it is quite weird that Bowser is so stupid, it almost feels like he wants to be beaten.
This is my stack. There are many like it, but this one is mine.My stack is my best friend. It is my life. I must master it as I must master my life.Without me, my stack is useless. Without my stack, I am useless. I must fire my requests true. I must shoot straighter than my hackers who are trying to kill me. I must shoot him before he shoots me. I will ...My stack is human, even as I am human, because it is my life. Thus, I will learn it as a brother. I will learn its weaknesses, its strength, its parts, its accessories, its ORMs and its asset bundlers. I will keep my stack clean and ready, even as I am clean and ready. We will become part of each other. We will ...Before God, I swear this creed. My stack and I are the defenders of my website. We are the masters of our enemy. We are the saviors of my life.So be it, until victory is mine and there is no enemy, but peace!
Explanation: this is an allusion to the Rifleman's Creed. This particular version talks about the website stack chosen for a website, i.e. the libraries used.
Ciro Santilli has always felt that choosing a stack is an almost religious choice. It is perhaps part of why the prayer style of the original Rifleman's Creed resonates with the web stack choice.
It is very hard to know how things are going go, the ups and downs, before putting big hours into it.
And once you start, it is hard, though not impossible, to move away.
The same allusion would make sense with any complex library choice, but it is particularly apparent in web development since there are so many different web stacks to choose from. A bit like rifles, they are all somewhat fungible, though of course not as much.