Upgrading a Fedora 39 Installation on a Btrfs Filesystem with an openSUSE Style Subvolume Layout to Fedora 40
Published: May 18, 2024, 9 p.m.
Updated: None
In a recent article on this site, a process to convert a Fedora 39 installation on a Btrfs filesystem with a simple subvolume layout to the hierarchical subvolume layout style of openSUSE with Snapper integration, automatic creation of snapshots, GRUB menu integration of snapshots, and other conveniences was presented. Since then Fedora 40 (and Ultramarine Linux 40) has been released. The installation and configuration presented in the earlier article survived a release upgrade to 40 with the Btrfs/Snapper configuration intact.
This article describes the release upgrade process using the CLI approach (although the native GUI package manager of the desketop environment used in the installation, KDE's Discover, is able to perform the release upgrade). More importantly, it describes the post-upgrade fixes to GRUB necessary, partly, because of Fedora's non-standard implementation of GRUB.
Introduction
Fedora is a highly refined and reliable distribution. As such, its package management infrastructure provides a reliable release upgrade experience, with either a command line tool, or any of the GUI package management utilities provided by its various editions, including KDE's Discover included with the Plasma Spin. I have used the upgrade tools in past Fedora editions without any issues, but I was not sure that the the upgrade tools would leave the Btrfs/Snapper/grub-btrfs configuration on a Fedora 39 (actually Ultramarine Linux 39) installation on a Btrfs filesystem that was converted to an openSUSE style Btrfs subvolume layout intact.
It did! However, GRUB packages had to be reinstalled and the GRUB firmware application installed to the ESP (EFI System Partition) had to be reinstalled.
Snapper Rollback Before Upgrade
Before upgrading to 40, I performed a rollback in order to make sure that the upgrade would modify a new snapshot subvolume that becomes the root of the filesystem hierarchy and not the initial snapshot subvolume to which contained the original installation. The following image illustrates the initial state of the system before the rollback and the rollback command and its output. The Konsole> window at the top shows the output of three commands that illustrates the state before the rollback:
snapper list
btrfs subvolume get-default /
and
cat /proc/cmdline
The first lists the existing snapshots available before the rollback. Note, for comparison with the state after the rollback, that the latest snapshot was number 189. The second shows that before the rollback, the default subvolume was
/@-0/.snapshots/1/snapshot,
the initial snapshot subvolume to which the original filesystem hierarchy root was moved by the process described in A Fedora Installation with an openSUSE Style Btrfs Subvolume Layout and Snapper Intergration for System Snapshots and Rollbacks, the 1 indicating Snapshot #1. And the third shows that before the rollback the kernel image path was in the default subvolume.
The middle Konsole window is displaying the GRUB menu confugration /boot/grub2/grub.cfg. This also shows that the kernel image and intial ram disk images are in the Snapshot #1.
In the bottom window, the Snapper rollback command is executed to rollback to Snapshot #189, a snapshot created at the last boot by the systemd timer snapper-boot.timer. The output shows that it created a read-only copy of the current, before rollback snapshot subvolume that is the current root of the filesystem hierarchy (Snapshot 190), and a read-write snapshot copy of the snapshot to which to rollback, Snapshot 191. It sets the default Btrfs subvolume as the new read-write snapshot, which will be the new root of the filesystem hierarchy and mounted automatically at boot by /etc/fstab (see A Fedora Installation with an openSUSE Style Btrfs Subvolume Layout and Snapper Intergration for System Snapshots and Rollbacks).
The State of the System Relevant to the Btrfs/Snapper Configuration Before Rollback
Before the rollback the root of the filesystem hierarchy was in Snapshot 1. After the rollback it will eb Snapshot 191.
After the rollback the new root of the filesystem hierarchy is in Snapshot 191, wheras before the rollback it was in Snapshot 1. The next image shows a Konsole window, at the left of the screen, with the same three commands as the topmeost window in the previous image. The output of the first command --
snapper list
indicates, by the * to the right of 191 that Snapshot 191 is the snapshot currently mounted at / and the snapshot that will be mounted at / at next boot (because it is the default subvolume). The second shows the kernel commadn line indicating that the kernel image was in Snapshot 191 with subvolume path /@-0/.snapshots/191/snapshot. The third ouput shows that the new default subvolume is /@-0/.snapshots/191/snapshot.
The State of the System Relevant to the Btrfs/Snapper Configuration After Rollback
After the rollback the root of the filesystem hierarchy was Snapshot 191, wheras before it was Snapshot 1.
When the system was rebooted after the rollback command, the GRUB menu did not reflect the new snapshot subvolume as the new system root. It was necessary to manually edit the default GRUB menu entry by pressing e on the menu item and then chainging the kernel and initial RAM disk image paths from /@-0/.snapshots/1/snapshot to /@-0/.snapshots/191/snapshot. The edited GRUB menu entry is shown in the following image.
Edited GRUB Menu Entry After Rollback
On the next boot after rollback it was necessary to edit the GRUB menu entry by pressing e on the menu item and changing the kernel and initial RAM disk image paths to the new snapshot subvolume that contains the system filessystem hierarchy root.
Then after the desktop was started, it was then necessary to update the GRUB menu configuration with
grub2-mkconfig -o /boot/grub2/grub.cfgso that the GRUB menu would reflect the new snapshot subvolume as the location of the new filesystem hierarchy root.
The Konsole windows at the right of the image show the GRUB menu configuration before the update (top windoow), the bottom of the output of the grub2-mkconfig command, which incorporates grub-btrfs actions (middle window), and the GRUB menu configuration after the update (bottom window). After the update, the GRUB menu configuration correctly looks for the kernel and intial RAM disk images at the new system root, in /@-0/.snapshots/191/snapshots, wheras before it had been at /@-0/.snapshots/1/snapshots (top window).
Release Upgrade
After the rollback, making Snapshot 191 the subvolume containing the root of the system filesystem hierarchy, instead of the original Snapshot 1, it was time to perfom the upgrade. The process steps are listed below. The Fedora Quick Docs page referenced below has details, including procedures for resolving errors caused by dependency issues, which were not necessary in my case.
-
Fully update the current system.
sudo dnf upgrade --refresh
-
Download the upgraded packages and prepare the upgrade transaction.
sudo dnf system-upgrade download --releasever=40
-
Start the upgrade. The actual upgrade will be offline after an automatic reboot.
sudo fng system-upgrade reboot
The following set of images illustrate the two types of user feedback provided by the upgrade tool during the upgrade. The first image shows the screen presented by default during the offline upgrade, indicating the progress graphically. Swithcing to one of the other virtual consoles by pressing Ctrl + Alt + FnX follows the messages created by the upgrade tool.
The Offline Update
The first image shows the screen presented by default during the offline upgrade, which indicates the progress. Swithcing to one of the other virtual consoles by pressing Ctrl + Alt + FnX follows the messages created by the upgrade tool.
Rebooting After Upgrade
The Btrfs/Snapper related aspects of the Fedora 39 system converted to the openSUSE style of subvolume layout with Snapper integration survived the upgrade. There was a challenge, however, with respect to the non-standard way Fedora uses GRUB, presumably to increase reliability in case there are problems. This non-standard feature is described in the second reference, but briefly, according to the reference, the GRUB configuration normally written to /boot/grub/grub.cfg, or more precisely, in Fedora's case, /boot/grub2/grub.cfg, is actually merged with another GRUB configuration file written to the Fedora directory in the ESP at /boot/efi/EFI/fedora/grub.cfg by the grub-common. The merged configuration files are what actually generates the GRUB menu. (In my case, with the integration of grub-btrfs -- see A Fedora Installation with an openSUSE Style Btrfs Subvolume Layout and Snapper Intergration for System Snapshots and Rollbacks -- grub-btrfs's own configuration is also merged to produce the GRUB menu item listing available snapshots.)
The following image shows two Konsole panes showing the contents of the directories which contain the two grub.cfg files. The left pane displays a tree listing of /boot/grub2/ and the right pane displays a tree listing of /boot/efi/EFI/fedora -- Fedora's directory in the EFI System Partition.
An Element of Fedora's Non-Standard Use of GRUB
The GRUB configuration actually used by Fedora's GRUB firmware application is generated by a merger of /boot/grub2/grub.cfg and another stored in the EFI System Partition, accessible in the system from /boot/efi/EFI/fedora. The image shows the contents of /boot/grub2/ and /boot/efi/EFI/fedora.
Comparing the GRUB configuration file /boot/grub2/grub.cfg, which contains the subvolume path from main top-level Btrfs subvolume, illustrated in various images in this article, and the GRUB configuration file in the ESP, /boot/efi/EFI/grub.cfg, it seems the only thing the latter adds to the final configuration is the value of the $prefix GRUB variable. This parameter informs GRUB of the device, i.e., the disk and partition, that contains the rest of the kernel and initial RAM disk image paths. Despite the apparent redundancy, GRUB could not provide a GRUB menu entry that points to the correct image path that reflects the current snapshot subvolume containing the root of the filesystem hierarchy, if the configuration file in the ESP did not have the correct path.
The grub.cfg in the ESP, which only gets updated when the GRUB firmware is reinstalled in the ESP (see next section), does not get updated by the grub2-mkconfig command, and as a result did not reflect the new snapshot subvolume after the rollback. So after the upgrade, even after updating the GRUB configuration after the rollback, the GRUB menu after the release upgrade did not reflect the correct kernel image or initial RAM disk image paths; it used the path /@-0/.snapshots/1/snapshot/boot/vmlinuz-xxx... instead of/@-0/.snapshots/191/snapshot/boot/vmlinuz-xxx.... I simply did not notice the problem because I immediately performed the upgrade after only confirming /boot/grub2/grub.cfg was updated to reflect paths in the new snapshot subvolume, and not the grub.cfg in the ESP. (This issue is corrected in the next section.)
The Btrfs/Snapper Related Characteristics of the Upgraded System
All Btrfs/Snapper relaed aspects of the system survived the upgrade. However, because, Fedora uses a duplicate of /boot/grub2/grub.cfg stored in the ESP, a non-standard use of GRUB, the first boot after upgrade required editing the kernel and initial RAM disk paths before booting into the correct snapshot subvolume.
As a result it was necessary to edit the GRUB menu entry manually, as discussed above, to reflect the correct snapshot subvolume in the kernel initial RAM disk paths in order to boot the system after the upgrade. After booting in this way, I confirmed that the Btrfs/Snapper configurations, and /boot/grub2/grub.cfg reflected the correct snapshot subvolume as containing the filesystem hierarchy root, as shown in the above image. The image shows that the default Btrfs subvolume is set to Snapshot 191, /boot/grub2/grub.cfg contains kernel and initial RAM disk paths that include Snapshot 191, and the kernel command line includes Snapshot 191. However, as described above, because the grub.cfg reflected the pre-rollback snapshot subvolume as that containing the root of the filesystem hierarchy, the GRUB configuration was not able to start the correct kernel image. This issue is corrected in the next section.
Post-Upgrade Fix for GRUB
After booting into the Fedora 40 (Ultramarine 40) system by editing the GRUB menu entry for reasons described above, and confirming that the Btrfs/Snapper configurations and /boot/grub2/grub.cfg file reflected the correct snapshot subvolume for the system root, it was necessary to update the duplicate GRUB configuration file in Fedora's directory in the ESP with the following process.
-
Delete the grub.cfg file in the ESP.
sudo rm -f /boot/efi/EFI/fedora/grub.cfg
-
Reinstall the GRUB packages grub2-efi, grub2-efi-modules, and several shim packages which can be specified to DNF as shim-\*. The command to install these, and the first part of its output is shown in the following listing:
brook on Ultramarine-16ITH6 ~ ❯ sudo dnf reinstall grub2-efi grub2-efi-modules shim-\* [sudo] password for brook: Fedora 40 - x86_64 - Updates 2.8 kB/s | 21 kB 00:07 Fedora 40 - x86_64 - Updates 25 kB/s | 2.8 MB 01:54 Terra 40 657 B/s | 1.0 kB 00:01 Terra 40 1.3 kB/s | 19 kB 00:14 Ultramarine Linux 40 628 B/s | 1.0 kB 00:01 Ultramarine Linux 40 2.6 kB/s | 16 kB 00:06 Dependencies resolved. ============================================================================================================ Package Architecture Version Repository Size ============================================================================================================ Reinstalling: grub2-efi-ia32 x86_64 1:2.06-121.fc40 updates 2.1 M grub2-efi-x64 x86_64 1:2.06-121.fc40 updates 2.1 M grub2-efi-x64-modules noarch 1:2.06-121.fc40 updates 1.1 M shim-ia32 x86_64 15.8-3 fedora 413 k shim-unsigned-x64 x86_64 15.8-2 fedora 460 k shim-x64 x86_64 15.8-3 fedora 467 k Transaction Summary ============================================================================================================ Total download size: 6.5 M Installed size: 24 M Is this ok [Y/n]:
image captionimage sub caption