Post 1 - Installing Linux is (Somewhat) Easier Than It Used to Be

Late last year, and earlier this year, I started thinking about whether I could continue using Debian Stable as my primary Linux distribution. Another thought which had been in my mind for a while was that I should try a rolling update distribution, such as Arch Linux. I never made the switch because Debian Stable works pretty well: It is extremely stable because all the packages are at least 3 years old, with security updates managed by the Debian team. (I have not idea whether they backport bug fixes; I have not run into any bug whose fix I wanted, so this did not affect me anyway.) But recently, the Linux desktop world has been moving quite fast: Wayland and PipeWire are becoming the norm (replacing X11 and PulseAudio). Old tools are being rewritten in Rust (such as ripgrep, fd, alacritty, autojump-rs) and the rewrites are much better than their older counterparts. These were some things that pushed me towards testing out Manjaro, a rolling update distribution based on Arch Linux. This post is about the installation process, not the OS itself, because I have only used it for a couple days at this point.

Installing most Linux distribution has become quite simple, compared to a few years ago. I believe this is because of two things: First, mostly thanks to the Calamares installer project, the installer for various distributions looks nearly identical. Calamares is a generic installer project, and they seem to be determined to make the installation process simple.

Second, this was the first time I was installing Linux after having understood a fair bit about UEFI and the boot process, during the past months. (I was in a UEFI rabbithole for a few weeks.) Understanding UEFI makes the installation process easy: I was no longer scared about my computer not booting after an installation, because I knew that I would be able to recover from it fairly easily. Even if the installation goes wrong, I know that I can configure the firmware that implements UEFI (and runs before the operating system) to boot from another boot entry or from a Live USB.

By controlling the BootOrder stored in Non-Volatile RAM¹ (NVRAM), either through the UEFI configuration interface or even changing values from a Linux live USB. As long as the BootOrder is correct in the NVRAM, and the boot entry for the operating system (Boot0002, say) points to a valid EFI executable file on the disk, the system will certainly boot and get to the Grub boot selection screen.

The following diagram is the “short” version of my understanding of UEFI:²

I also started noticing some things during the installation process itself: Immediately after you select the partitioning and click Continue, the partition table is updated and the new partitions are created as you want them. Further, these partitions are mounted where they would usually be, but with a prefix. For instance, the boot partition is mounted at /installer/boot, the EFI System Partition (ESP) is mounted at /installer/boot/efi, and the root partition is mounted at /installer (or something like this: I did not record the command output during installation, and by the time it dawned on me that I should have, the installation was already complete. If you are going through a Manjaro installation, please tell me the prefix that you see.)

Once the installation is complete, before rebooting the computer and leaving the live USB environment, you can check the three changes that the installer would have made.

Updates to the NVRAM

The NVRAM is updated with a path to the new EFI binary. There is a UUID in the output of efibootmgr, and this is the UUID of the /boot/efi partition, which you selected in the partitioning step of the installer. For Manjaro, this is a binary file and not really readable, which is a shame. When installing Debian, the file is called shimx64.efi and there is a Grub configuration besides the file, which is a readable text file.

$ sudo efibootmgr | grep -i manjaro
Boot0002* Manjaro	HD(5,GPT,PARTUUID-OF-ESP,0x800,0xfa000)/\EFI\Manjaro\grubx64.efi

$ sudo lsblk -o uuid,partuuid,mountpoint | grep PARTUUID-OF-ESP
ABCD-EFGH                            PARTUUID-OF-ESP /boot/efi

$ sudo file /boot/efi/EFI/Manjaro/grubx64.efi
/boot/efi/EFI/Manjaro/grubx64.efi: PE32+ executable for EFI (application), x86-64 (stripped to external PDB), 4 sections

I think that the installer can do better here: I selected a 260 MB FAT partition which was already present on my disk and marked it with the boot flag and mounted it at /boot/efi. There were two issues:

First, it gave me a rather scary warning telling me that I had not selected an ESP partition that was atleast 300 MB and that I may not be able to boot into the system: This is incorrect. The ESP being at least 300 MB is a recommendation from Manjaro, not a requirement for the system to boot (if you update kernels a lot and don’t clean up the old initrd files, then the size of /boot can become a problem.) However, this has nothing to do with the ability to boot. I guess if there isn’t enough space in the existing /boot/efi the installation would fail.

Secondly, it did not tell me that I should mount the existing ESP at /boot/efi. If no partition’s mountpoint is set to /boot/efi at the partitioning step, I believe that the EFI binary will be written to the disk at /boot/efi, which is inside the /boot partition, and thus not something that the UEFI firmware will be able to boot from: The firmware that implements UEFI will boot from any ESP and ESP only supports FAT.

This would cause the OS to be installed and Grub, the Kernel, and initrd to exist in /boot and the system itself to exist in the root partition, but there would be no way to boot into it. This might be the reason for many issues that users face. I found a thread about it on the Manjaro forums, and I wanted to reply to it, but the thread is closed! So, users will forever keep running into this issue and thread and never be able to find the solution except through trial and error:

100% confirmed now. The missing esp flag is making me unable to boot Manjaro after install, it doesn`t even detect as a bootable system.

– Manjaro Forum

Grub, kernel, and initrd inside /boot

The other partitoin that you must have created in the partitioning step would have been the /boot partition. This partition typically has a copy of Grub, Grub’s configuration, the Linux kernel (vmlinuz-*.), and the initramfs image, which is a bootstrap system that runs right before the Kernel itself runs (if I am not wrong).

This would be a typical ext4 partition. So, it is easy to just mount it and check what is inside the partition. It can be mounted using applications such as gnome-disks if you don’t want to use the mount CLI tool.

Updates to /etc/crypttab and /etc/fstab

If you are installing Linux after 2020, you are hopefully using an encrypted LUKS partition for your root and home partitions, or you might be using a LUKS-based logical volume which contains both the root and home partitions, as this makes it easier to resize your partitions. Nevertheless, in both cases, Grub itself needs to know how to unlock the root partition. In modern Grub versions, I see these sort of options for the Linux kernel start up command:

search --no-floppy --fs-uuid --set=root UUID-OF-BOOT-EXT4-PARTITION

linux	/vmlinuz-6.12-x86_64 root=UUID=UUID-OF-DECRYPTED-ROOT-EXT4-PARTITION rw  quiet cryptdevice=UUID=UUID-OF-ENCRYPTED-LUKS-PARTITION:luks-UUID-OF-ENCRYPTED-LUKS-PARTITION root=/dev/mapper/luks-UUID-OF-ENCRYPTED-LUKS-PARTITION splash apparmor=1 security=apparmor resume=UUID=UUID-OF-SWAP-PARTITION udev.log_priority=3

I don’t fully understand the options here. Particularly, I don’t understand why the root= option is repeated twice. Once before cryptdevice= and once after. I guess only the second one applies? It is also interesting to see that it is “resuming” the Swap partition. What is that supposed to mean? I am planning to convert my currently unencrypted Swap partition to a cryptswap, which is a plain LUKS partition with a random key (because I don’t care about the data that is written there after a reboot). These are some things to look into later!

References

The following article and the UEFI specification helped me a lot in undertsanding how UEFI really works in practice:

• UEFI boot: how does that actually work, then? - AdamW on Linux and more
• 2. Overview — UEFI Specification 2.11 documentation