by Ciro Santilli (@cirosantilli, 32)
Ciro's tutorial: Linux Kernel Module Cheat.
Ciro's word of caution for 2019 aspiring system programmers: Should you waste your life with systems programming?
This is basically a direct consequence of backward design.
The higher the level you can operate at, the better.
C is better than assembly, userland better than kerneland.
The ideal level to operate at, and one of humankind's greatest ambitions is "AGI, make me money", the highest possible level.
Only go down a level when it seems necessary.
Magic software that allows you to write a single program that runs on a wide range of hardware.
Bare metal programming is to run a program without an operating system below it.
Or in other words, it is basically implementing an operating system/firmware yourself ad hoc, together with your actual program.
Zephyr is cool. Its installation setup is annoying. But the project is cool.
Real hardware is for newbs. Real hardware is for newbs.
Tested on Ubuntu 23.10 we approximately follow instructions from: docs.zephyrproject.org/3.4.0/develop/getting_started/index.html stopping before the "Flash the sample" section, as we don't flash QEMU. We just run it.
sudo apt install --no-install-recommends git cmake ninja-build gperf \
  ccache dfu-util device-tree-compiler wget \
  python3-dev python3-pip python3-setuptools python3-tk python3-wheel xz-utils file \
  make gcc gcc-multilib g++-multilib libsdl2-dev libmagic1
python3 -m venv ~/zephyrproject/.venv
source ~/zephyrproject/.venv/bin/activate
pip install west
west init ~/zephyrproject
cd ~/zephyrproject
west update
west zephyr-export
cd ~
wget https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.16.1/zephyr-sdk-0.16.1_linux-x86_64.tar.xz
tar xvf zephyr-sdk-0.16.1_linux-x86_64.tar.xz
cd zephyr-sdk-0.16.1
./setup.sh
The installation procedure install all compiler toolchains for us, so we can then basically compile for any target. It also fetches the latest Git source code of Zephyr under: 
~/zephyrproject/zephyr
The "most default" blinky hello world example which blinks an LED is a bit useless for us because QEMU doesn't have LEDs, so instead we are going to use one of the UART examples which will print characters we can see on QEMU stdout.
Let's start with the hello world example on an x86 target: 
cd ~/zephyrproject/zephyr
west build -b qemu_x86 samples/hello_world -t run
and it outputs: 
Hello World! qemu_x86
The qemu_x64 on the output comes from the CONFIG_BOARD macro github.com/zephyrproject-rtos/zephyr/blob/c15ff103001899ba0321b2c38013d1008584edc0/samples/hello_world/src/main.c#L11 
#include <zephyr/kernel.h>

int main(void)
{
	printk("Hello World! %s\n", CONFIG_BOARD);
	return 0;
}
You can also first cd into the directory that you want to build in to avoid typing samples/hello_world all the time: 
cd ~/zephyrproject/zephyr/samples/hello_world
zephyr west build -b qemu_x86 -t run
You can also build and run separately with: 
west build -b qemu_x86
west build -t run
Another important option is: 
west build -t menuconfig
But note that it does not modify your prj.conf automatically for you.
Let's try on another target: 
rm -rf build
zephyr west build -b qemu_cortex_a53 -t run
and same output, but on a completely different board! The qemu_cortex_a53 board is documented at: docs.zephyrproject.org/3.4.0/boards/arm64/qemu_cortex_a53/doc/index.html
The list of all examples can be seen under: 
ls ~/zephyrproject/zephyr/samples
which for example contains: 
zephyrproject/zephyr/samples/hello_world
So run another sample simply select it, e.g. to run zephyrproject/zephyr/samples/synchronization: 
west build -b qemu_cortex_a53 samples/synchronization -t run
It ain't perfect, but it's decent enough.
From a technical point of view, it can do anything that Microsoft Windows can. Except being forcefully installed on every non-MacOS 2019 computer you can buy.
Ciro Santilli's conversion to Linux happened around 2012, and was a central part of Ciro Santilli's Open Source Enlightenment, since it fundamentally enables the discovery and contribution to open source software. Because what awesome open source person would waste time porting their amazing projects to closed source OSes?
Ciro's modest nature can be seen as he likes to compare this event Buddha's Great Renunciation.
Particularly interesting in the history of Linux is how it won out over the open competitors that were coming up in the time: MINIX (see the chat) and BSD Operating System that got legally bogged down at the critical growth moment.
Figure 1. xkcd 619: Supported Features. Source. This perfectly illustrates Linux development. First features that matter. Then useless features.
Video 1. Bill Gates vs Steve Jobs by Epic Rap Battles of History (2012) Source. Just stop whatever you are doing, and watch this right now. "I'm on Linux, bitch, I thought you GNU". Fandom explanations. It is just a shame that the Bill Gates actor looks absolutely nothing like the real gates. Actually, the entire Gates/Jobs parts are good, but not genial. But the Linux one is.
The fact that this foundation has a bunch of paid, closed, certification courses makes Ciro Santilli not respect them at all. They should be making open access content instead!
Documents the Linux kernel. Somewhat of a competitor to Linux Kernel Module Cheat, but more wordy and less automated.
As of 2020, no one knows how to build the major desktop distros fully from source into the ISO, and especially so in a reproducible build way. Everything is done in build servers somewhere with complicated layers of prebuilds. It's crap.
merlijn.sebrechts.be/blog/2020-08-02-why-one-snap-store/ has some very good comments on how snap is more closed than Flatpak.
However, many, many, many terrible horrors come with it:
The video folder is DCIM/Camera.
Respect. Big respect. Those people are hardcore from scratch hackers, and their wiki is amazing: wiki.archlinux.org/
It's just that Buildroot is more hardcore ;-)
But can you build the ISO full from source: Linux distribution buildable from source
Buildroot is good.
This thing is sexy.
Ciro Santilli's Linux distro of choice as of 2019.
It ain't perfect, but it's decent enough.
The greatest advantage of it being that it has the likely largest desktop user base, and therefore the highest likelihood that your problems are solved on Ask Ubuntu, and goes together with Ciro's philosophy that "people should do everything in the same way to factor stuff out", especially the open source losers.
Ciro considers that the killer flaw of Ubuntu, and most desktop distros of 2020, is that no one under the Sun knows how to build them fully from source: Linux distribution buildable from source. This is why Ciro based the Linux Kernel Module Cheat on Buildroot, see also: Linux distribution buildable from source.
On Ubuntu 23.10, a crash led to the creation of: 
/var/crash/_usr_bin_gnome-shell.1000.crash
After that simply running apport-cli as: 
apport-cli gnome-shell
led to the creation of: bugs.launchpad.net/ubuntu/+source/gnome-shell/+bug/2049368 a bug on the gnome-shell package.
Their crash system does not have an amazing user interface.
Tested on Ubuntu 21.10.
After something crashes, look under /var/crash for a crash file, which helps to determine which package to report under on Launchpad.
E.g. a file /var/crash/_usr_sbin_gdm3.0.crash makes you want to file the bug under gdm at: bugs.launchpad.net/ubuntu/+source/gdm/+filebug
Then, while reporting the bug, you want to give the developpers access to that .crash file. But you can't publicly upload it because it contains memory dumps and could contain secret information. The way to do it is to look at the ID under: 
sudo cat /var/crash/_usr_sbin_gdm3.0.uploaded
Ubuntu's crash report system has already uploaded the .crash for you, so you just have to confirm it and give the ID on the ticket.
You can view a list of all your uploaded errors at: 
xdg-open https://errors.ubuntu.com/user/$(sudo cat /var/lib/whoopsie/whoopsie-id)
and each of those contain a link to: 
https://errors.ubuntu.com/oops/<.uloaded error id>
which you yourself cannot see.
Running: 
sudo apport-unpack /var/crash/_usr_sbin_gdm3.0.crash /tmp/app
splits it up into a few files, but does not make any major improvements.
apport-retrace 
sudo apt install apport-retrace
sudo chmod 666 /var/crash/_usr_sbin_gdm3.0.crash
apport-retrace -g /var/crash/_usr_sbin_gdm3.0.crash
opens GDB with the core dump. Debug symbols are supplied as separate packages, which is a really cool idea: so you should be able to download them after the crash to see symbols. askubuntu.com/questions/487222/how-to-install-debug-symbols-for-installed-packages mentions how to install them. Official docs at: wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages
Tried: 
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
echo -e "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse\ndeb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt install ubuntu-dbgsym-keyring
but then sudo apt update fails with: 
E: The repository 'http://ddebs.ubuntu.com impish-security Release' does not have a Release file.
Had this happen on P14s on Ubuntu 23.10 while causally using Chromium. The screen went blank for a few seconds, but it apparently managed to reboot itself, and things started working again, except that and most windows were killed: 
[drm:gfx_v11_0_priv_reg_irq [amdgpu]] *ERROR* Illegal register access in command stream
[drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=5774109, emitted seq=5774111
[drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process chrome pid 14023 thread chrome:cs0 pid 14087
amdgpu 0000:64:00.0: amdgpu: GPU reset begin!
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[drm:gfx_v11_0_cp_gfx_enable.isra.0 [amdgpu]] *ERROR* failed to halt cp gfx
Dec 27 15:03:38 ciro-p14s kernel: amdgpu 0000:64:00.0: amdgpu: MODE2 reset
Dec 27 15:03:38 ciro-p14s kernel: amdgpu 0000:64:00.0: amdgpu: GPU reset succeeded, trying to resume
Dec 27 15:03:38 ciro-p14s kernel: [drm] PCIE GART of 512M enabled (table at 0x0000008000900
It appears to be a bug in the AMDGPU open source driver.
I think this was on Wayland. Possibly relatd but on X Window System, crashed the UI, showed message "oh no! Something has gone wrong." 
2024-01-13_21-55-07@ciro@ciro-p14s$ cat /var/log/apport.log
ERROR: apport (pid 975172) 2024-01-13 21:41:02,087: host pid 3528 crashed in a separate mount namespace, ignoring
INFO: apport (pid 975227) 2024-01-13 21:41:02,398: called for pid 2728, signal 5, core limit 0, dump mode 1
INFO: apport (pid 975227) 2024-01-13 21:41:02,401: executable: /usr/bin/gnome-shell (command line "/usr/bin/gnome-shell")
INFO: apport (pid 975227) 2024-01-13 21:41:12,667: wrote report /var/crash/_usr_bin_gnome-shell.1000.crash
Happened on P14s on Ubuntu 23.10, which started with fresh Ubuntu 23.10 install.
However it did not happen on Lenovo ThinkPad P51 (2017) also on Ubuntu 23.10 which had been upgraded several times from God knows what starting point... At first one had X11 (forced by Nvidia drivers) and the other Wayland, but moving to p14s X11 changed nothing.
Both were running GNOME Display Manager.
Same happens with Super + L, but also CLI commands: askubuntu.com/questions/7776/how-do-i-lock-the-desktop-screen-via-command-line
Switching to the other installed kernel, 5.9 made boot work.
The solution on kernel 6.2 was: 
sudo apt instal nvidia-driver-515
as per comments under: bugs.launchpad.net/ubuntu/+source/linux/+bug/2012559. This also made the nvidia driver work: Find GPU information in Ubuntu.
Previously I had: 
nvidia-driver-510
and it blew up before reaching disk decryption.
I also tried: 
nvidia-driver-525
but that broke in a different way: 
Finished apport-autoreport.service - Process error reports when automatic reporting is enabled.
nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000947d:0:0:407
(1 of 2) Job systemd-backlight@backlight: nvidia_e.service/start running (32s no limit)
GDM crashes sometimes when switching windows right after opening a new window: bugs.launchpad.net/ubuntu/+source/gdm/+bug/1956299
Does not happen every time, only some times. Can't figure out why. Usually happens when has suspended for a longer time.
bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-470/+bug/1946303 sounds like a likely report, Nvidia driver version 470, but can't find those error messages anywhere. The last line of: 
journalctl -o short-precise -k -b -1
once was: 
PM: suspend entry (deep)
which is when sleep starts.
This suggests that it is not a video bug then, seems that it is not waking up at all? Gotta try to SSH into it. OK. I did SSH into it, and that was fine, so it is just the video that won't start.
PM: suspend exit
bugs.launchpad.net/ubuntu/+source/linux/+bug/1949977 is another possible bug, based on kernel version. I'm running 5.13, which is one of the failing versions on the report. Can't find any interesting dmesg though.
In another crash: 
journalctl -o short-precise -k -b -1
had the following interesting lines: 
nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.
[24307.640014] NVRM: GPU at PCI:0000:01:00: GPU-18af74bb-7c72-ff70-e447-87d48378ea20
[24307.640018] NVRM: Xid (PCI:0000:01:00): 79, pid=8828, GPU has fallen off the bus.
[24307.640021] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
[24328.054022] nvidia-modeset: ERROR: GPU:0: The requested configuration of display devices (LGD (DP-4)) is not supported on this GPU.
[repeats several more times]
[24328.056767] nvidia-modeset: ERROR: GPU:0: The requested configuration of display devices (LGD (DP-4)) is not supported on this GPU.
[24328.056951] nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000927c:0:0:0x0000000f
[24328.056955] nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000927c:1:0:0x0000000f
[24328.056959] nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000927c:2:0:0x0000000f
[24328.056962] nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000927c:3:0:0x0000000f
[24328.056983] nvidia-modeset: ERROR: GPU:0: DP-4: Failed to disable DisplayPort audio stream-0
[24328.056992] nvidia-modeset: ERROR: GPU:0: Failed to query display engine channel state: 0x0000947d:0:0:0x0000000f
and there was a corresponding /var/crash/_usr_sbin_gdm3.0.crash.
Sometimes it just feels like Ubuntu devs don't actually use Ubuntu as a desktop.
Some extremelly anoying problems are introduced and just never get fixed, even though they feel so obvious!
Would never happen on Mac...
This BS started after the move to ZFS. The temporary solution appears to be: askubuntu.com/questions/1293685/out-of-space-on-boot-zpool-and-cant-run-updates-anymore/1374204#1374204
And then this to disable automatic snapshots in the future: askubuntu.com/questions/1233049/disable-automatic-zsys-snapshots-zfs-on-root/1279593#1279593 
sudo mv /etc/apt/apt.conf.d/90_zsys_system_autosnapshot /etc/apt/apt.conf.d/90_zsys_system_autosnapshot.disabled
This has annoyed Ciro Santilli for many years, it is just too wasteful of screen space on laptops!
Or likely more generally, on GNOME desktop, which is the default desktop environment as of Ubuntu 22.10.
Legal issues stalled them at the turning point of the Internet, and Linux won. Can't change history.
Did Apple just fork it and made Mac OS X without giving anything back?
Non-POSIX only here.
If you are a programmer, grep becomes a verb: "to grep" means "to search text files", much like "to Google" means "to search random stuff online".
The OS that the Gods ordered be made.
One is reminded of Ulillillia, see also: www.youtube.com/watch?v=9-79yOZ13qg The Story of Ulillillia by Atrocity Guide (2019)
Video 1. I like elephants and God likes elephants. Source.
For a quick and dirty introduction to the format, see: ELF Hello World Tutorial.
Introductory analysis of a simple example of the executable and Linkable Format.
ELF is the dominating file format for Linux. It competes with Mach-O for OS X and PE for Windows.
ELF supersedes .coff, which supersedes a.out.
The LSB basically links to other standards with minor extensions, in particular:
A handy summary can be found at: 
man elf
Spin like mad between:
The ELF standard specifies multiple file formats:
  • Object files (.o).
    Intermediate step to generating executables and other formats: 
    Source code

    |
    | Compilation
    |
    v

Object file

    |
    | Linking
    |
    v

Executable
    Object files exist to make compilation faster: with make, we only have to recompile the modified source files based on timestamps.
    We have to do the linking step every time, but it is much less expensive.
  • Executable files (no standard Linux extension).
    This is what the Linux kernel can actually run.
  • Archive files (.a).
    Libraries meant to be embedded into executables during the Linking step.
  • Shared object files (.so).
    Libraries meant to be loaded when the executable starts running.
  • Core dumps.
    Such files may be generated by the Linux kernel when the program does naughty things, e.g. segfault.
    They exist to help debugging the program.
In this tutorial, we consider only object and executable files.
  • Compiler toolchains generate and read ELF files.
    Sane compilers should use a separate standalone library to do the dirty work. E.g., Binutils uses BFD (in-tree and canonical source).
  • Operating systems read and run ELF files.
    Kernels cannot link to a library nor use the C stlib, so they are more likely to implement it themselves.
    This is the case of the Linux kernel 4.2 which implements it in th file fs/binfmt_elf.c.
It is non-trivial to determine what is the smallest legal ELF file, or the smaller one that will do something trivial in Linux.
In this example we will consider a saner hello world example that will better capture real life cases.
Let's break down a minimal runnable Linux x86-64 example:
hello_world.asm
section .data
    hello_world db "Hello world!", 10
    hello_world_len  equ $ - hello_world
section .text
    global _start
    _start:
        mov rax, 1
        mov rdi, 1
        mov rsi, hello_world
        mov rdx, hello_world_len
        syscall
        mov rax, 60
        mov rdi, 0
        syscall
Compiled with: 
nasm -w+all -f elf64 -o 'hello_world.o' 'hello_world.asm'
ld -o 'hello_world.out' 'hello_world.o'
TODO: use a minimal linker script with -T to be more precise and minimal.
Versions:
  • NASM 2.10.09
  • Binutils version 2.24 (contains ld)
  • Ubuntu 14.04
We don't use a C program as that would complicate the analysis, that will be level 2 :-)
Running: 
hd hello_world.o
gives: 
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  01 00 3e 00 01 00 00 00  00 00 00 00 00 00 00 00  |..>.............|
00000020  00 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000030  00 00 00 00 40 00 00 00  00 00 40 00 07 00 03 00  |....@.....@.....|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000080  01 00 00 00 01 00 00 00  03 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 02 00 00 00 00 00 00  |................|
000000a0  0d 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000b0  04 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000000c0  07 00 00 00 01 00 00 00  06 00 00 00 00 00 00 00  |................|
000000d0  00 00 00 00 00 00 00 00  10 02 00 00 00 00 00 00  |................|
000000e0  27 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |'...............|
000000f0  10 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000100  0d 00 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
00000110  00 00 00 00 00 00 00 00  40 02 00 00 00 00 00 00  |........@.......|
00000120  32 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |2...............|
00000130  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000140  17 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
00000150  00 00 00 00 00 00 00 00  80 02 00 00 00 00 00 00  |................|
00000160  a8 00 00 00 00 00 00 00  05 00 00 00 06 00 00 00  |................|
00000170  04 00 00 00 00 00 00 00  18 00 00 00 00 00 00 00  |................|
00000180  1f 00 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 00  30 03 00 00 00 00 00 00  |........0.......|
000001a0  34 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |4...............|
000001b0  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  27 00 00 00 04 00 00 00  00 00 00 00 00 00 00 00  |'...............|
000001d0  00 00 00 00 00 00 00 00  70 03 00 00 00 00 00 00  |........p.......|
000001e0  18 00 00 00 00 00 00 00  04 00 00 00 02 00 00 00  |................|
000001f0  04 00 00 00 00 00 00 00  18 00 00 00 00 00 00 00  |................|
00000200  48 65 6c 6c 6f 20 77 6f  72 6c 64 21 0a 00 00 00  |Hello world!....|
00000210  b8 01 00 00 00 bf 01 00  00 00 48 be 00 00 00 00  |..........H.....|
00000220  00 00 00 00 ba 0d 00 00  00 0f 05 b8 3c 00 00 00  |............<...|
00000230  bf 00 00 00 00 0f 05 00  00 00 00 00 00 00 00 00  |................|
00000240  00 2e 64 61 74 61 00 2e  74 65 78 74 00 2e 73 68  |..data..text..sh|
00000250  73 74 72 74 61 62 00 2e  73 79 6d 74 61 62 00 2e  |strtab..symtab..|
00000260  73 74 72 74 61 62 00 2e  72 65 6c 61 2e 74 65 78  |strtab..rela.tex|
00000270  74 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |t...............|
00000280  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  01 00 00 00 04 00 f1 ff  |................|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002b0  00 00 00 00 03 00 01 00  00 00 00 00 00 00 00 00  |................|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 03 00 02 00  |................|
000002d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002e0  11 00 00 00 00 00 01 00  00 00 00 00 00 00 00 00  |................|
000002f0  00 00 00 00 00 00 00 00  1d 00 00 00 00 00 f1 ff  |................|
00000300  0d 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000310  2d 00 00 00 10 00 02 00  00 00 00 00 00 00 00 00  |-...............|
00000320  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000330  00 68 65 6c 6c 6f 5f 77  6f 72 6c 64 2e 61 73 6d  |.hello_world.asm|
00000340  00 68 65 6c 6c 6f 5f 77  6f 72 6c 64 00 68 65 6c  |.hello_world.hel|
00000350  6c 6f 5f 77 6f 72 6c 64  5f 6c 65 6e 00 5f 73 74  |lo_world_len._st|
00000360  61 72 74 00 00 00 00 00  00 00 00 00 00 00 00 00  |art.............|
00000370  0c 00 00 00 00 00 00 00  01 00 00 00 02 00 00 00  |................|
00000380  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000390
Running: 
hd hello_world.out
gives: 
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  02 00 3e 00 01 00 00 00  b0 00 40 00 00 00 00 00  |..>.......@.....|
00000020  40 00 00 00 00 00 00 00  10 01 00 00 00 00 00 00  |@...............|
00000030  00 00 00 00 40 00 38 00  02 00 40 00 06 00 03 00  |....@.8...@.....|
00000040  01 00 00 00 05 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 40 00 00 00 00 00  00 00 40 00 00 00 00 00  |..@.......@.....|
00000060  d7 00 00 00 00 00 00 00  d7 00 00 00 00 00 00 00  |................|
00000070  00 00 20 00 00 00 00 00  01 00 00 00 06 00 00 00  |.. .............|
00000080  d8 00 00 00 00 00 00 00  d8 00 60 00 00 00 00 00  |..........`.....|
00000090  d8 00 60 00 00 00 00 00  0d 00 00 00 00 00 00 00  |..`.............|
000000a0  0d 00 00 00 00 00 00 00  00 00 20 00 00 00 00 00  |.......... .....|
000000b0  b8 01 00 00 00 bf 01 00  00 00 48 be d8 00 60 00  |..........H...`.|
000000c0  00 00 00 00 ba 0d 00 00  00 0f 05 b8 3c 00 00 00  |............<...|
000000d0  bf 00 00 00 00 0f 05 00  48 65 6c 6c 6f 20 77 6f  |........Hello wo|
000000e0  72 6c 64 21 0a 00 2e 73  79 6d 74 61 62 00 2e 73  |rld!...symtab..s|
000000f0  74 72 74 61 62 00 2e 73  68 73 74 72 74 61 62 00  |trtab..shstrtab.|
00000100  2e 74 65 78 74 00 2e 64  61 74 61 00 00 00 00 00  |.text..data.....|
00000110  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000150  1b 00 00 00 01 00 00 00  06 00 00 00 00 00 00 00  |................|
00000160  b0 00 40 00 00 00 00 00  b0 00 00 00 00 00 00 00  |..@.............|
00000170  27 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |'...............|
00000180  10 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  21 00 00 00 01 00 00 00  03 00 00 00 00 00 00 00  |!...............|
000001a0  d8 00 60 00 00 00 00 00  d8 00 00 00 00 00 00 00  |..`.............|
000001b0  0d 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001c0  04 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001d0  11 00 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 00 00 00 00 00 00  e5 00 00 00 00 00 00 00  |................|
000001f0  27 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |'...............|
00000200  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000210  01 00 00 00 02 00 00 00  00 00 00 00 00 00 00 00  |................|
00000220  00 00 00 00 00 00 00 00  90 02 00 00 00 00 00 00  |................|
00000230  08 01 00 00 00 00 00 00  05 00 00 00 07 00 00 00  |................|
00000240  08 00 00 00 00 00 00 00  18 00 00 00 00 00 00 00  |................|
00000250  09 00 00 00 03 00 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 00 00  98 03 00 00 00 00 00 00  |................|
00000270  4c 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |L...............|
00000280  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002a0  00 00 00 00 00 00 00 00  00 00 00 00 03 00 01 00  |................|
000002b0  b0 00 40 00 00 00 00 00  00 00 00 00 00 00 00 00  |..@.............|
000002c0  00 00 00 00 03 00 02 00  d8 00 60 00 00 00 00 00  |..........`.....|
000002d0  00 00 00 00 00 00 00 00  01 00 00 00 04 00 f1 ff  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000002f0  11 00 00 00 00 00 02 00  d8 00 60 00 00 00 00 00  |..........`.....|
00000300  00 00 00 00 00 00 00 00  1d 00 00 00 00 00 f1 ff  |................|
00000310  0d 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000320  00 00 00 00 04 00 f1 ff  00 00 00 00 00 00 00 00  |................|
00000330  00 00 00 00 00 00 00 00  2d 00 00 00 10 00 01 00  |........-.......|
00000340  b0 00 40 00 00 00 00 00  00 00 00 00 00 00 00 00  |..@.............|
00000350  34 00 00 00 10 00 02 00  e5 00 60 00 00 00 00 00  |4.........`.....|
00000360  00 00 00 00 00 00 00 00  40 00 00 00 10 00 02 00  |........@.......|
00000370  e5 00 60 00 00 00 00 00  00 00 00 00 00 00 00 00  |..`.............|
00000380  47 00 00 00 10 00 02 00  e8 00 60 00 00 00 00 00  |G.........`.....|
00000390  00 00 00 00 00 00 00 00  00 68 65 6c 6c 6f 5f 77  |.........hello_w|
000003a0  6f 72 6c 64 2e 61 73 6d  00 68 65 6c 6c 6f 5f 77  |orld.asm.hello_w|
000003b0  6f 72 6c 64 00 68 65 6c  6c 6f 5f 77 6f 72 6c 64  |orld.hello_world|
000003c0  5f 6c 65 6e 00 5f 73 74  61 72 74 00 5f 5f 62 73  |_len._start.__bs|
000003d0  73 5f 73 74 61 72 74 00  5f 65 64 61 74 61 00 5f  |s_start._edata._|
000003e0  65 6e 64 00                                       |end.|
000003e4
An ELF file contains the following parts:
  • ELF header. Points to the position of the section header table and the program header table.
  • Section header table (optional on executable). Each has e_shnum section headers, each pointing to the position of a section.
  • N sections, with N <= e_shnum (optional on executable)
  • Program header table (only on executable). Each has e_phnum program headers, each pointing to the position of a segment.
  • N segments, with N <= e_phnum (only on executable)
The order of those parts is not fixed: the only fixed thing is the ELF header that must be the first thing on the file: Generic docs say:
Although the figure shows the program header table immediately after the ELF header, and the section header table following the sections, actual files may differ. Moreover, sections and segments have no specified order. Only the ELF header has a fixed position in the file.
In pictures: sample object file with three sections: 
            +-------------------+
            | ELF header        |---+
+---------> +-------------------+   | e_shoff
|           |                   |<--+
| Section   | Section header 0  |
|           |                   |---+ sh_offset
| Header    +-------------------+   |
|           | Section header 1  |---|--+ sh_offset
| Table     +-------------------+   |  |
|           | Section header 2  |---|--|--+
+---------> +-------------------+   |  |  |
            | Section 0         |<--+  |  |
            +-------------------+      |  | sh_offset
            | Section 1         |<-----+  |
            +-------------------+         |
            | Section 2         |<--------+
            +-------------------+
But nothing (except sanity) prevents the following topology: 
            +-------------------+
            | ELF header        |---+ e_shoff
            +-------------------+   |
            | Section 1         |<--|--+
+---------> +-------------------+   |  |
|           |                   |<--+  | sh_offset
| Section   | Section header 0  |      |
|           |                   |------|---------+
| Header    +-------------------+      |         |
|           | Section header 1  |------+         |
| Table     +-------------------+                |
|           | Section header 2  |---+            | sh_offset
+---------> +-------------------+   | sh_offset  |
            | Section 2         |<--+            |
            +-------------------+                |
            | Section 0         |<---------------+
            +-------------------+
But some newbies may prefer PNGs :-)
Figure 1. ELF Executable and Linkable Format diagram by Ange Albertini. Source.
We will get into more detail later, but it is good to have it in mind now:
  • section: exists before linking, in object files.
    One ore more sections will be put inside a single segment by the linker.
    Major information sections contain for the linker: is this section:
    • raw data to be loaded into memory, e.g. .data, .text, etc.
    • or metadata about other sections, that will be used by the linker, but disappear at runtime e.g. .symtab, .srttab, .rela.text
  • segment: exists after linking, in the executable file.
    Contains information about how each segment should be loaded into memory by the OS, notably location and permissions.
Running: 
readelf -h hello_world.o
outputs: 
Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class:                             ELF64
Data:                              2's complement, little endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       0
Type:                              REL (Relocatable file)
Machine:                           Advanced Micro Devices X86-64
Version:                           0x1
Entry point address:               0x0
Start of program headers:          0 (bytes into file)
Start of section headers:          64 (bytes into file)
Flags:                             0x0
Size of this header:               64 (bytes)
Size of program headers:           0 (bytes)
Number of program headers:         0
Size of section headers:           64 (bytes)
Number of section headers:         7
Section header string table index: 3
Running: 
readelf -h hello_world.out
outputs: 
Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class:                             ELF64
Data:                              2's complement, little endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       0
Type:                              EXEC (Executable file)
Machine:                           Advanced Micro Devices X86-64
Version:                           0x1
Entry point address:               0x4000b0
Start of program headers:          64 (bytes into file)
Start of section headers:          272 (bytes into file)
Flags:                             0x0
Size of this header:               64 (bytes)
Size of program headers:           56 (bytes)
Number of program headers:         2
Size of section headers:           64 (bytes)
Number of section headers:         6
Section header string table index: 3
Bytes in the object file: 
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  01 00 3e 00 01 00 00 00  00 00 00 00 00 00 00 00  |..>.............|
00000020  00 00 00 00 00 00 00 00  40 00 00 00 00 00 00 00  |........@.......|
00000030  00 00 00 00 40 00 00 00  00 00 40 00 07 00 03 00  |....@.....@.....|
Executable: 
00000000  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00  |.ELF............|
00000010  02 00 3e 00 01 00 00 00  b0 00 40 00 00 00 00 00  |..>.......@.....|
00000020  40 00 00 00 00 00 00 00  10 01 00 00 00 00 00 00  |@...............|
00000030  00 00 00 00 40 00 38 00  02 00 40 00 06 00 03 00  |....@.8...@.....|
Structure represented: 
# define EI_NIDENT 16

typedef struct {
    unsigned char   e_ident[EI_NIDENT];
    Elf64_Half      e_type;
    Elf64_Half      e_machine;
    Elf64_Word      e_version;
    Elf64_Addr      e_entry;
    Elf64_Off       e_phoff;
    Elf64_Off       e_shoff;
    Elf64_Word      e_flags;
    Elf64_Half      e_ehsize;
    Elf64_Half      e_phentsize;
    Elf64_Half      e_phnum;
    Elf64_Half      e_shentsize;
    Elf64_Half      e_shnum;
    Elf64_Half      e_shstrndx;
} Elf64_Ehdr;
Manual breakdown:
  • 0 0: EI_MAG = 7f 45 4c 46 = 0x7f 'E', 'L', 'F': ELF magic number
  • 0 4: EI_CLASS = 02 = ELFCLASS64: 64 bit elf
  • 0 5: EI_DATA = 01 = ELFDATA2LSB: little endian data
  • 0 6: EI_VERSION = 01: format version
  • 0 7: EI_OSABI (only in 2003 Update) = 00 = ELFOSABI_NONE: no extensions.
  • 0 8: EI_PAD = 8x 00: reserved bytes. Must be set to 0.
  • 1 0: e_type = 01 00 = 1 (big endian) = ET_REl: relocatable format
    On the executable it is 02 00 for ET_EXEC.
    Another important possibility for the executable is ET_DYN for PIE executables and shared libraries.
    ET_DYN tells the Linux kernel that the code is position independent, and can loaded at a random memory location with ASLR.
  • 1 2: e_machine = 3e 00 = 62 = EM_X86_64: AMD64 architecture
  • 1 4: e_version = 01 00 00 00: must be 1
  • 1 8: e_entry = 8x 00: execution address entry point, or 0 if not applicable like for the object file since there is no entry point.
    On the executable, it is b0 00 40 00 00 00 00 00. The kernel puts the RIP directly on that value when executing. It can be configured by the linker script or -e. But it will segfault if you set it too low: stackoverflow.com/questions/2187484/why-is-the-elf-execution-entry-point-virtual-address-of-the-form-0x80xxxxx-and-n
  • 2 0: e_phoff = 8x 00: program header table offset, 0 if not present.
    40 00 00 00 on the executable, i.e. it starts immediately after the ELF header.
  • 2 8: e_shoff = 40 7x 00 = 0x40: section header table file offset, 0 if not present.
  • 3 0: e_flags = 00 00 00 00 Arch specific. i386 docs say:
    The Intel386 architecture defines no flags; so this member contains zero.
  • 3 4: e_ehsize = 40 00: size of this elf header. TODO why this field needed? Isn't the size fixed?
  • 3 6: e_phentsize = 00 00: size of each program header, 0 if not present.
    38 00 on executable: it is 56 bytes long
  • 3 8: e_phnum = 00 00: number of program header entries, 0 if not present.
    02 00 on executable: there are 2 entries.
  • 3 A: e_shentsize and e_shnum = 40 00 07 00: section header size and number of entries
  • 3 E: e_shstrndx (Section Header STRing iNDeX) = 03 00: index of the .shstrtab section.
Array of Elf64_Shdr structs.
Each entry contains metadata about a given section.
e_shoff of the ELF header gives the starting position, 0x40 here.
e_shentsize and e_shnum from the ELF header say that we have 7 entries, each 0x40 bytes long.
So the table takes bytes from 0x40 to 0x40 + 7 + 0x40 - 1 = 0x1FF.
Some section names are reserved for certain section types: www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html#special_sections e.g. .text requires a SHT_PROGBITS type and SHF_ALLOC + SHF_EXECINSTR
Running: 
readelf -S hello_world.o
outputs: 
There are 7 section headers, starting at offset 0x40:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align
  [ 0]                   NULL             0000000000000000  00000000
       0000000000000000  0000000000000000           0     0     0
  [ 1] .data             PROGBITS         0000000000000000  00000200
       000000000000000d  0000000000000000  WA       0     0     4
  [ 2] .text             PROGBITS         0000000000000000  00000210
       0000000000000027  0000000000000000  AX       0     0     16
  [ 3] .shstrtab         STRTAB           0000000000000000  00000240
       0000000000000032  0000000000000000           0     0     1
  [ 4] .symtab           SYMTAB           0000000000000000  00000280
       00000000000000a8  0000000000000018           5     6     4
  [ 5] .strtab           STRTAB           0000000000000000  00000330
       0000000000000034  0000000000000000           0     0     1
  [ 6] .rela.text        RELA             0000000000000000  00000370
       0000000000000018  0000000000000018           4     2     4
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), l (large)
  I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)
The struct represented by each entry is: 
typedef struct {
    Elf64_Word  sh_name;
    Elf64_Word  sh_type;
    Elf64_Xword sh_flags;
    Elf64_Addr  sh_addr;
    Elf64_Off   sh_offset;
    Elf64_Xword sh_size;
    Elf64_Word  sh_link;
    Elf64_Word  sh_info;
    Elf64_Xword sh_addralign;
    Elf64_Xword sh_entsize;
} Elf64_Shdr;
Contained in bytes 0x40 to 0x7F.
The first section is always magic: www.sco.com/developers/gabi/2003-12-17/ch4.sheader.html says:
If the number of sections is greater than or equal to SHN_LORESERVE (0xff00), e_shnum has the value SHN_UNDEF (0) and the actual number of section header table entries is contained in the sh_size field of the section header at index 0 (otherwise, the sh_size member of the initial entry contains 0).
There are also other magic sections detailed in Figure 4-7: Special Section Indexes.
In index 0, SHT_NULL is mandatory. Are there any other uses for it: stackoverflow.com/questions/26812142/what-is-the-use-of-the-sht-null-section-in-elf ?

Ancestors

  1. Software
  2. Computer
  3. Information technology
  4. Area of technology
  5. Technology
  6. Index

Incoming links

Discussion (0)

Sign up or sign in create discussions.

There are no discussions about this article yet.

View article source