Creating / deleting drivers’ home directories is not cleanly/reliably implemented

[ClydeByrdIII]

I'm prefixing this by saying this is all from a perspective of linux environment with kernels 6.8 with ubuntu distro.

Based on

container-libs/storage/drivers/overlay/overlay.go

Line 226 in e70c309

func checkAndRecordOverlaySupport(home, runhome string) (bool, error) {

If overlay support feature isn't already cached on the host (which will happen every boot since it caches in /run/containers/storage/overlay, backed by tmpfs)
It will try to test support, on failure it'll delete home which in default configurations would be /var/lib/containers/storage/overlay if I'm not mistaken.

But on success, it'll also attempt to delete home, before it eventually recreates it

container-libs/storage/drivers/overlay/overlay.go

Lines 709 to 716 in e70c309

	defer func() {
	// Permitted to fail, since the various subdirectories
	// can be empty or not even there, and the home might
	// legitimately be not empty
	_ = unix.Unmount(mergedDir, unix.MNT_DETACH)
	_ = os.RemoveAll(layerDir)
	_ = os.Remove(home)
	}()

In the XFS project quota libs section it's suggested that we can assign a project id to /var/lib/containers/storage/overlay

container-libs/storage/drivers/quota/projectquota_supported.go

Lines 112 to 125 in e70c309

	// xfs_quota tool can be used to assign a project id to the driver home directory, e.g.:
	// echo 100000:/var/lib/containers/storage/overlay >> /etc/projects
	// echo 200000:/var/lib/containers/storage/volumes >> /etc/projects
	// echo storage:100000 >> /etc/projid
	// echo volumes:200000 >> /etc/projid
	// xfs_quota -x -c 'project -s storage volumes' /<xfs mount point>
	//
	// In the example above, the storage directory project id will be used as a
	// "start offset" and all containers will be assigned larger project ids
	// (e.g. >= 100000). Then the volumes directory project id will be used as a
	// "start offset" and all volumes will be assigned larger project ids
	// (e.g. >= 200000).
	// This is a way to prevent xfs_quota management from conflicting with
	// containers/storage.

But, if the home deletion is successful, doesn't that remove the XFS project quota ID on home if the admin set one up before running a workload?

[mtrmac]

Thanks for reaching out.

Well, are you actually seeing that happen? I see at least one reason why that directory should never be removed . (related: #666 might break that eventually).

Note to self: The history of that logic, removing the driver’s home, is probably 76b3aa8 . And on the other hand, #680 (comment) discusses other reasons why this design is not ideal.

[ClydeByrdIII]

Ah you're right. I figured out what was stripping it in my environment. Sorry for the bother.

FWIW to confirm your suspicion I spun up a brand new VM that's never ran podman and ran strace on it.

I can see that it makes l and .has-mount-program under /var/lib/containers/storage/overlay

and never removes them so that os.Remove(home) would always fail it seems.

Seems a little tricky, but gets the ideal result for me for now.

[mtrmac]

I do think this is a problematic design — .has-mount-program is not always set, and l might be going away; so this is something we should clean up; I was just thinking it’s unlikely to manifest yet.

Reopening to track this situation.