install: Enable installing to multi device parents by ckyrouac · Pull Request #1911 · bootc-dev/bootc

ckyrouac · 2026-01-14T19:54:35Z

When the root filesystem spans multiple backing devices (e.g., LVM across multiple disks), discover all parent devices and find ESP partitions on each. For bootupd/GRUB, install the bootloader to all devices with an ESP partition, enabling boot from any disk in a multi-disk setup. systemd-boot and zipl only support single-device configurations.

This adds a new integration test validating both single-ESP and dual-ESP multi-device scenarios.

Fixes: #481

Assisted-by: Claude Code (Opus 4.5)

gemini-code-assist

Code Review

This pull request successfully enables installing to multi-device parent filesystems, such as LVM spanning multiple disks. It correctly discovers all parent devices and, for bootupd/GRUB, installs the bootloader to all devices with an ESP partition. For bootloaders that only support single-device configurations like systemd-boot and zipl, the implementation correctly defaults to using the first available device. The changes are well-architected, adapting data structures and logic to handle multiple devices. A new, thorough integration test validates both single and dual ESP scenarios. Overall, this is a solid enhancement with good error handling and logging. I have one suggestion to further improve the robustness of ESP detection.

crates/lib/src/install.rs

crates/lib/src/bootloader.rs

crates/lib/src/bootc_composefs/boot.rs

ckyrouac · 2026-01-15T20:26:46Z

waiting to merge until the patch release goes out

crates/lib/src/bootc_composefs/boot.rs

HuijingHei · 2026-01-21T08:39:46Z

Maybe this PR could help to consider #1916 ? @ckyrouac

crates/lib/src/bootc_composefs/boot.rs

crates/lib/src/bootloader.rs

crates/lib/src/install.rs

HuijingHei · 2026-03-17T07:59:06Z

tmt/tests/booted/test-multi-device-esp.nu

+    # See https://tmt.readthedocs.io/en/stable/stories/features.html#reboot-during-test
+    match $env.TMT_REBOOT_COUNT? {
+        null | "0" => test_single_esp,
+        "1" => { test_dual_esp; test_three_devices_partial_esp; tmt-reboot },


How about add another test scenario like RAID1 that using the whole disks (see coreos/bootupd#1059), for example:

root@localhost-live:/home/fedora# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS vda 253:0 0 30G 0 disk └─md126 9:126 0 30G 0 raid1 ├─md126p1 259:0 0 477M 0 part ├─md126p2 259:1 0 954M 0 part /boot └─md126p3 259:2 0 28.6G 0 part / vdb 253:16 0 30G 0 disk └─md126 9:126 0 30G 0 raid1 ├─md126p1 259:0 0 477M 0 part ├─md126p2 259:1 0 954M 0 part /boot └─md126p3 259:2 0 28.6G 0 part /

To create RAID1 using command:
sudo mdadm -CR /dev/md126 -e 1 -l1 -n 2 /dev/loop0 /dev/loop1 --assume-clean

crates/lib/src/bootc_composefs/boot.rs

cgwalters · 2026-04-01T17:45:09Z

crates/lib/src/bootloader.rs

+/// files before reinstallation.  On multi-device setups only the first
+/// ESP is mounted and cleaned; stale files on additional ESPs are left
+/// in place (bootupd will overwrite them during installation).
+// TODO: clean all ESPs on multi-device setups


Hmm yeah maybe should move this logic into bootupd

cgwalters · 2026-04-01T17:46:06Z

hack/provision-derived.sh

+    *) copr_distro="centos-stream" ;;
+esac
+# Update bootc from rhcontainerbot copr; the new bootupd
+# requires a newer bootc than what ships in some base images.


But wait we're building this as part of our CI here right?

yea, this is definitely sub-optimal. This is in place to unblock the bootupd install which will fail due to the bootc version being <= 1.14.1. I might be able to move the bootupd copr install to later in the Dockerfile, after the new bootc build is installed...

hack/provision-derived.sh

cgwalters · 2026-04-01T22:25:13Z

OK I pushed a few cleanups here, when you get back tomorrow can you look @ckyrouac ?

ckyrouac · 2026-04-02T20:29:14Z

OK I pushed a few cleanups here, when you get back tomorrow can you look @ckyrouac ?

lgtm, fixed a doc string bug and removed the composefs commit.

Three issues prevented a successful Anaconda-based bootc install onto a RAID 1 array with per-disk ESP partitions: 1. Bind mount failure during ostreecontainer payload install: Anaconda mounts the root FS at /mnt/sysimage and then tries to bind-mount /mnt/sysimage/boot/efi2 and /boot/efi3 into /mnt/sysroot. However, it does not create the mount point directories for the secondary ESPs before the bind mount, causing: mount: /mnt/sysroot/boot/efi2: special device /mnt/sysimage/boot/efi2 does not exist. Fix: add a %pre-install script that creates the directories after the root FS is mounted but before the payload runs. 2. Empty secondary ESPs after installation: Anaconda only installs the bootloader (shim, GRUB, grub.cfg) to the primary ESP at /boot/efi. The secondary ESPs (efi2, efi3) are formatted and mounted but receive no bootloader files. Additionally, the 'noauto' fsoption causes Anaconda to unmount them before %post. Fix: add a %post --nochroot script that explicitly mounts each secondary ESP by device path and copies the full EFI directory tree from the primary ESP. 3. GRUB cannot find the kernel on the RAID root filesystem: The GRUB EFI binary shipped with CentOS Stream 10 does not include the mdraid1x module. GRUB's grub.cfg uses 'search --fs-uuid' to locate the root filesystem and load the kernel from the ostree deployment path, but it cannot access the md array. Fix: add a separate /boot partition (ext4, 1G) on the primary disk. GRUB can read ext4 natively without mdraid support. The kernel and initramfs live on /boot, and the initramfs handles RAID assembly for the root filesystem during boot. Note: /boot is only on vda (not mirrored), so only disk 1 can boot independently. Full boot redundancy from any disk would require either whole-disk RAID (where GRUB reads from the raw mirror) or a GRUB build with mdraid1x support. This is a known limitation tracked in: - bootc-dev/bootc#1911 - coreos/bootupd#1077

cgwalters · 2026-04-06T14:17:41Z

Needs a rebase 🏄 otherwise looks sane to me!

Assisted-by: Claude Code (Opus 4) Signed-off-by: ckyrouac <ckyrouac@redhat.com> Signed-off-by: Colin Walters <walters@verbum.org>

The composefs BLS and UKI boot setup paths called find_partition_of_esp() directly on the device, which fails when the root filesystem is on an LVM logical volume (the ESP is on the parent disk, not the LV). The store module had the same issue via require_single_root() + find_partition_of_esp(). Switch all call sites to find_colocated_esps() which walks up to the physical disk(s) via find_all_roots() before searching for the ESP, consistent with what install_systemd_boot and mount_esp_part already do. Assisted-by: Claude Code (Opus 4) Signed-off-by: ckyrouac <ckyrouac@redhat.com> Signed-off-by: Colin Walters <walters@verbum.org>

The test was using `get_target_image` which returns the upstream `docker://quay.io/centos-bootc/centos-bootc:stream10`. On composefs+grub variants provisioned with an updated bootupd from copr, the upstream image has stock bootupd with incompatible EFI update metadata, causing the install to fail with "Failed to find EFI update metadata". Switch to using `containers-storage:localhost/bootc` (the locally-built image), matching the pattern used by test-32, test-37, and test-38. The locally-built image has the updated bootupd with compatible metadata. Assisted-by: Claude Code (Opus 4) Signed-off-by: ckyrouac <ckyrouac@redhat.com> Signed-off-by: Colin Walters <walters@verbum.org>

The initial change to use locally-built images had two additional issues on composefs: 1. containers-storage: transport fails on composefs's read-only root with "mkdir /.local: read-only file system". Fix by exporting the image to an OCI layout directory on writable /var/tmp instead. 2. run_install() was masking /sysroot/ostree and removing bootupd update metadata, which composefs needs for bootloader installation and boot binaries. Fix by making run_install() skip these ostree-specific workarounds on composefs systems. Note: the composefs install-outside-container code path still has a separate bug ("Shared boot binaries not found" in boot.rs:745) that needs fixing in the Rust code. Assisted-by: Claude Code (Opus 4) Signed-off-by: ckyrouac <ckyrouac@redhat.com> Signed-off-by: Colin Walters <walters@verbum.org>

The multi-device ESP test creates ESP partitions and expects bootupd to install a UEFI bootloader. On BIOS-booted systems, bootupd instead tries to install GRUB for i386-pc, which requires a BIOS Boot Partition and fails. The test plan already requests UEFI provisioning via the hardware hint, but Testing Farm does not always honor this on CentOS Stream x86_64. Add a runtime check for /sys/firmware/efi so the test skips gracefully on BIOS hosts rather than failing. Assisted-by: Claude Code (Opus 4.6) Signed-off-by: ckyrouac <ckyrouac@redhat.com> Signed-off-by: Colin Walters <walters@verbum.org>

Extract the repeated PATH environment variable string into a set_default_path() method on BwrapCmd. The bwrap environment may not have a complete PATH, causing tools like bootupctl or sfdisk to not be found. This consolidates the workaround into one place. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

Several improvements to ESP partition discovery: Add find_partition_of_esp_optional() returning Result<Option<&Device>> to cleanly separate three outcomes: found, absent, and genuinely unexpected errors (like unsupported partition table types). The existing find_partition_of_esp() is now a thin wrapper that converts None to Err. Add find_first_colocated_esp() helper to replace a 10-line pattern that was repeated verbatim 5 times across boot.rs and store/mod.rs. Deduplicate roots in find_all_roots() using a seen-set: in complex topologies like multipath, multiple parent branches can converge on the same physical disk. find_colocated_esps() now uses the optional variant to properly propagate real errors while treating absence normally. Also extract the match-on-if-else in setup_composefs_bls_boot into a let binding for readability. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org> Signed-off-by: Chris Kyrouac <ckyrouac@redhat.com>

The no-ESP test only checked for a non-zero exit code, which would also pass if podman itself failed for unrelated reasons. Check that the output contains "ESP" to confirm the right failure mode. Assisted-by: OpenCode (Claude Opus 4) Signed-off-by: Colin Walters <walters@verbum.org>

jmarrero

lgtm

jmarrero · 2026-04-06T23:22:10Z

s390x builds have been broken for a while too, and we know about the ZIPL/systemd boot limitations no. Maybe we should add those to the docs in a follow up.

github-actions bot added the area/install Issues related to `bootc install` label Jan 14, 2026

bootc-bot bot requested a review from cgwalters January 14, 2026 19:54

gemini-code-assist bot reviewed Jan 14, 2026

View reviewed changes

crates/lib/src/install.rs Outdated Show resolved Hide resolved

cgwalters previously approved these changes Jan 15, 2026

View reviewed changes

crates/lib/src/bootloader.rs Outdated Show resolved Hide resolved

crates/lib/src/bootc_composefs/boot.rs Outdated Show resolved Hide resolved

crates/lib/src/bootc_composefs/boot.rs Outdated Show resolved Hide resolved

ckyrouac dismissed cgwalters’s stale review via f7b1892 January 15, 2026 20:26

ckyrouac force-pushed the multiple-backing-devices branch from f2a175a to f7b1892 Compare January 15, 2026 20:26

ckyrouac marked this pull request as draft January 15, 2026 20:26

ckyrouac force-pushed the multiple-backing-devices branch 3 times, most recently from 77b65cb to d03c6fa Compare January 16, 2026 17:31

cgwalters reviewed Jan 20, 2026

View reviewed changes

crates/lib/src/bootc_composefs/boot.rs Outdated Show resolved Hide resolved

cgwalters force-pushed the multiple-backing-devices branch from d03c6fa to 9b1c313 Compare January 21, 2026 13:08

ckyrouac force-pushed the multiple-backing-devices branch 2 times, most recently from 6802697 to 081f3b2 Compare January 22, 2026 15:29

ckyrouac marked this pull request as ready for review January 22, 2026 19:28

bootc-bot bot requested a review from jeckersb January 22, 2026 19:28

ckyrouac force-pushed the multiple-backing-devices branch from 081f3b2 to 9d0e284 Compare January 23, 2026 14:53

cgwalters requested changes Jan 26, 2026

View reviewed changes

crates/lib/src/bootc_composefs/boot.rs Outdated Show resolved Hide resolved

crates/lib/src/bootloader.rs Outdated Show resolved Hide resolved

crates/lib/src/install.rs Outdated Show resolved Hide resolved

cgwalters mentioned this pull request Jan 26, 2026

lib/install: Use mounted ESP for install to-filesystem #1953

Open

This was referenced Feb 11, 2026

Blockdev cleanup #2000

Merged

Blockdev bugfixes #2005

Merged

cgwalters mentioned this pull request Feb 12, 2026

Failure on RAID1 bootloader update coreos/bootupd#1059

Open

ckyrouac marked this pull request as draft February 16, 2026 17:12

ckyrouac force-pushed the multiple-backing-devices branch from 9d0e284 to 4ccc192 Compare March 11, 2026 19:40

HuijingHei reviewed Mar 17, 2026

View reviewed changes

ckyrouac force-pushed the multiple-backing-devices branch 2 times, most recently from 541691d to baa6c67 Compare March 24, 2026 19:17

ckyrouac requested a review from cgwalters March 24, 2026 19:17

ckyrouac force-pushed the multiple-backing-devices branch 4 times, most recently from 9da140b to b7d4d61 Compare March 26, 2026 13:37

cgwalters previously approved these changes Apr 1, 2026

View reviewed changes

cgwalters mentioned this pull request Apr 1, 2026

composefs: Open deployments dir from physical root, not ambient path #2115

Closed

cgwalters dismissed their stale review via b245dc6 April 1, 2026 22:23

cgwalters force-pushed the multiple-backing-devices branch 2 times, most recently from b245dc6 to 52602b4 Compare April 1, 2026 22:24

ckyrouac force-pushed the multiple-backing-devices branch 2 times, most recently from 174ba1c to 3fcf28c Compare April 2, 2026 20:27

ckyrouac force-pushed the multiple-backing-devices branch 2 times, most recently from 06f8406 to 282160b Compare April 3, 2026 14:27

ckyrouac added 5 commits April 6, 2026 10:27

install: Enable installing to devices with multiple parents

dd979a4

Assisted-by: Claude Code (Opus 4) Signed-off-by: ckyrouac <ckyrouac@redhat.com> Signed-off-by: Colin Walters <walters@verbum.org>

ckyrouac force-pushed the multiple-backing-devices branch from 282160b to cf0ea08 Compare April 6, 2026 14:40

cgwalters added 3 commits April 6, 2026 14:18

ckyrouac force-pushed the multiple-backing-devices branch from cf0ea08 to 64e815e Compare April 6, 2026 18:18

jmarrero approved these changes Apr 6, 2026

View reviewed changes

jmarrero merged commit 3ec055f into bootc-dev:main Apr 6, 2026
61 of 65 checks passed

Conversation

ckyrouac commented Jan 14, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ckyrouac commented Jan 15, 2026

Uh oh!

Uh oh!

HuijingHei commented Jan 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HuijingHei Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgwalters Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

cgwalters Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ckyrouac Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cgwalters commented Apr 1, 2026

Uh oh!

ckyrouac commented Apr 2, 2026

Uh oh!

cgwalters commented Apr 6, 2026

Uh oh!

jmarrero left a comment

Choose a reason for hiding this comment

Uh oh!

jmarrero commented Apr 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants