storage: add splitfdstream for efficient layer transfer with reflink support by giuseppe · Pull Request #651 · containers/container-libs

giuseppe · 2026-02-10T12:32:04Z

Add a new splitfdstream package that enables efficient container layer transfer between storage instances by separating tar metadata from file content. File content is passed as file descriptors.

mtrmac

6% of the code base for an extremely niche use case :/

I’m afraid I can’t spend time on this now; and anyway some other work on pkg/archive needs to happen first, and expect the rebase to be somewhat non-trivial.

giuseppe · 2026-02-10T15:55:34Z

6% of the code base for an extremely niche use case :/

I’m afraid I can’t spend time on this now; and anyway some other work on pkg/archive needs to happen first, and expect the rebase to be somewhat non-trivial.

what other work would block this feature?

I don't love either the amount of extra code, but it is almost all in new APIs/packages that are experimental and not leaking too much to the rest of the library.

We went around this problem for a long time, and in the end this seems like an ~ok solution to not expose too much of the containers storage internals augmenting the tar format that is already plugged everywhere.

giuseppe · 2026-02-10T17:43:19Z

to add more context: the goal of this feature is to be able to copy a layer, similarly to what we would do with Diff, but having the possibility to deduplicate files using reflinks. We need two separate copies because OS files can't be kept on the containers storage for different reasons, a podman system reset -f should not break the system, and OS files need to be relabeled.

There are other interesting use cases enabled with a RPC like facilitate usage from different languages (Rust in our case), inject into a build container to grab R/O access to layers/images, but this is out of scope now.

I realize complexity is not trivial just to get access to the raw files, but what alternatives do we have?

From the store API PoV we could just extend it to return a map $LAYER_FILE -> $PATH_TO_THE_FILE, but how would we use it from Rust?

Any suggestions?

mtrmac · 2026-02-10T18:24:55Z

(Just to avoid a misunderstanding, I didn’t really read the PR. I have, at this point, ~no opinion on RPC vs. Go API.)

cgwalters · 2026-02-10T19:18:18Z

storage/pkg/archive/archive.go

+	return &tarReaderIterator{
+		tr:     tr,
+		trBuf:  trBuf,
+		buffer: make([]byte, 1<<20),


Seems like 1<<20 should be in a const somewhere with a rationale?

cgwalters · 2026-02-10T19:19:02Z

storage/pkg/chrootarchive/splitfdstream_linux.go

+
+// splitFDStreamSocketDescriptor is the fd for the Unix socket used to
+// receive file descriptors via SCM_RIGHTS in the re-exec child.
+const splitFDStreamSocketDescriptor = 5


Hardcoding this seems a bit fragile, can't we pass it as an env var or a CLI arg?

cgwalters · 2026-02-10T19:20:11Z

storage/pkg/chrootarchive/splitfdstream_linux.go

+// FDs are streamed to the child process one-at-a-time over a Unix socket
+// using SCM_RIGHTS, avoiding EMFILE from inheriting too many FDs at exec.


It doesn't have to be truly one at a time, see also https://github.com/cgwalters/jsonrpc-fdpass?tab=readme-ov-file#41-fd-batching

In my initial PoC work here I handled this by just sending the splitfdstream inline data itself as a fd (could be O_TMPFILE or memfd or pipe), and then sending all the associated fds over jsonrpc-fdpass which automatically handles buffering.

Ensure we processing just 1 recvmsg() at a time (only ~200 fds on Linux) seems really reasonable and unlikely to get anywhere close to modern fd limits.

So if we do choose jsonrpc-fdpass, I think we also need to standardize this "splitfdstream serialization".

cgwalters · 2026-02-10T19:30:29Z

storage/pkg/fileutils/reflink_linux.go

+		return err
+	}
+
+	// Try copy_file_range - kernel-level copy, more efficient than userspace


Surprising we weren't doing this before.

cgwalters · 2026-02-10T19:33:57Z

storage/pkg/splitfdstream/fdpass.go

+
+// ReadLine reads bytes until a newline is encountered, using recvmsg.
+// Any FDs received are closed since line-reading doesn't expect them.
+func (p *FDPasser) ReadLine() ([]byte, error) {


Why would one want this?

cgwalters · 2026-02-10T19:35:02Z

storage/pkg/splitfdstream/protocol.go

+
+// JSON-RPC 2.0 protocol constants
+const (
+	JSONRPCVersion = "2.0"


BTW see also https://github.com/cgwalters/jsonrpc-fdpass-go

giuseppe · 2026-02-11T16:13:53Z

still PoC, so you can avoid a review as it is not yet ready but we can use to analyze the complexity.

Moved to use https://github.com/cgwalters/jsonrpc-fdpass-go

Now the code size is almost 50% than the last revision

packit-as-a-service · 2026-02-11T16:46:06Z

Packit jobs failed. @containers/packit-build please check.

TomSweeneyRedHat · 2026-03-10T19:17:59Z

Per @mtrmac, this should merge before #677 does

mtrmac · 2026-03-10T19:23:28Z

No, #677 would be infrastructure for this PR. I really mean that I have little opinion on merging this PR, and that without that, #677 is not clearly an improvement.

giuseppe · 2026-03-10T20:10:22Z

this needs to be rebased on top of #677 anyway, I'll need to rework it on top of that

TomSweeneyRedHat · 2026-03-18T21:51:02Z

@giuseppe looks like a rebase may be needed here.

giuseppe · 2026-03-23T07:57:10Z

@kolyshkin are these changes to the tar package a problem for the work you are doing there?

giuseppe · 2026-03-27T10:47:39Z

@kolyshkin can you PTAL?

cgwalters · 2026-04-06T15:54:48Z

storage/drivers/overlay/overlay_splitfdstream.go

+	}
+	defer tarStream.Close()
+
+	// Write splitfdstream data directly to a memfd to avoid buffering in memory.


This type of comment I find a bit distracting because it's almost circular - a memfd is memory too...how about just:

A memfd is a convenient way to buffer the metadata

Or maybe just no comment at all. Dunno.

Here and in other places like

// Set up ID mappings idMappings := options.IDMappings

The comment basically adds nothing.

cgwalters · 2026-04-06T15:57:05Z

storage/drivers/overlay/overlay_splitfdstream.go

+	return streamFile, fds, nil
+}
+
+// convertTarToSplitFDStream converts a tar stream to a splitfdstream by parsing


This obviously works but it seems pretty inefficient to do - i.e. to read the entire contents of large file into memory and just discard it.

I guess we're doing this because the current API is "give a tarball" and this just builds on that without changing anything more fundamental?

cgwalters · 2026-04-06T15:57:29Z

storage/drivers/overlay/overlay_splitfdstream.go

+				if buf == nil {
+					buf = make([]byte, archive.CopyBufferSize)
+				}
+				// File not found in diff directory (e.g., naiveDiff was used),


When would the file not be found?

cgwalters · 2026-04-06T15:58:47Z

storage/drivers/overlay/overlay_splitfdstream.go

+				// Write a single prefix for the total size, then stream
+				// data in chunks.  The reader expects exactly one prefix
+				// per file entry.
+				if err := writer.WriteInlinePrefix(header.Size); err != nil {


Would perhaps be more elegant if writer.WriteInline returned a sub-io.Write impl then we could just do io.Copy to that?

cgwalters · 2026-04-06T16:06:28Z

common/pkg/json-proxy/handler.go

 		rb, err = h.GetLayerInfoPiped(ctx, req.Args)
 	case "FinishPipe":
 		rb, err = h.FinishPipe(ctx, req.Args)
+	case "GetSplitFDStreamSocket":


It seems like this may grow into a general replacement for the bespoke proxy API right? How about calling it

OpenJsonRPCFdPass ?

cgwalters · 2026-04-06T16:09:30Z

storage/pkg/splitfdstream/server_linux.go

+	"runtime"
+	"sync"
+
+	fdpass "github.com/cgwalters/jsonrpc-fdpass-go"


This can use https://github.com/bootc-dev/jsonrpc-fdpass-go now

bootc-dev/jsonrpc-fdpass-go#7

cgwalters · 2026-04-06T16:11:08Z

storage/pkg/splitfdstream/server_linux.go

+	// The library limits to MaxFDsPerMessage per sendmsg, so remaining
+	// FDs are sent as follow-up "fds" notifications.


Hmm the jsonrpc-fdpass protocol specifies to send more fds as whitespace but it's possible the Go implementation isn't doing this right

bootc-dev/jsonrpc-fdpass-go#4

cgwalters · 2026-04-06T16:11:32Z

storage/pkg/splitfdstream/types.go

+// findManifest finds the image manifest from BigData keys.
+// It looks for manifest-* keys containing an image manifest (has "config" field),
+// filtering out manifest lists/indexes.
+func findManifest(store Store, imageID string) ([]byte, error) {


Surely there's a common API for this?

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Extend the store with splitfdstream capabilities exposed via a UNIX socket for JSON-RPC communication. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Implement the SplitFDStreamDriver interface for the overlay driver, enabling efficient layer operations with reflink support. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

Add a new json-proxy method that returns a Unix socket file descriptor to the client. The client then speaks the jsonrpc-fdpass protocol directly over that socket for splitfdstream operations, bypassing the json-proxy for bulk data transfer. Bump protocolVersion to "0.2.9". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

cgwalters · 2026-04-08T13:59:26Z

I think this is overall OK as is, however - just to cross-reference from composefs/composefs-rs#228 (comment) - I think we should get a bit more ambitious and try adding a full new proxy API based on jsonrpc-fdpass instead. There's a lot of warts and bugs we could fix at the same time (like the bad idea of auto-converting v2s2 to OCI and other things from the historical PRs there).

That would then make the architecture much clearer: There's a jsonrpc-fdpass API for containers-storage:. We rework https://github.com/bootc-dev/containers-image-proxy-rs/ to support this (it'd be a relatively thin wrapper on top of jsonrpc-fdpass, just with stronger Rust types basically) and then it becomes very straightforward to plug in all of the existing places.

Just to also xref again I think this API has the potential to solve problems like containers/skopeo#658 (comment) too. But of course, that's where the choice of IPC becomes a big deal.

giuseppe · 2026-04-08T14:17:17Z

I think this is overall OK as is, however - just to cross-reference from composefs/composefs-rs#228 (comment) - I think we should get a bit more ambitious and try adding a full new proxy API based on jsonrpc-fdpass instead. There's a lot of warts and bugs we could fix at the same time (like the bad idea of auto-converting v2s2 to OCI and other things from the historical PRs there).

isn't that something we can do incrementally? I've tried to keep this as small as possible (e.g. dropped write support) to facilitate merging it, but I think we should get that back and let the copy code handle this as a shortcut for more efficient copies to different stores

github-actions bot added the storage Related to "storage" package label Feb 10, 2026

This was referenced Feb 10, 2026

[WIP] composefs-splitfdstream: new crate composefs/composefs-rs#228

Draft

reflink by default (if possible) when copying between containers-storage: instances #144

Open

giuseppe force-pushed the splitfdstream branch 2 times, most recently from 4cca2a3 to 3b6f591 Compare February 10, 2026 14:59

mtrmac reviewed Feb 10, 2026

View reviewed changes

giuseppe force-pushed the splitfdstream branch from 3b6f591 to 26cd4b6 Compare February 10, 2026 16:01

cgwalters reviewed Feb 10, 2026

View reviewed changes

giuseppe force-pushed the splitfdstream branch 2 times, most recently from 23ed544 to 8d74f84 Compare February 11, 2026 16:07

giuseppe force-pushed the splitfdstream branch from 8d74f84 to 51a6c01 Compare February 11, 2026 16:25

giuseppe force-pushed the splitfdstream branch 2 times, most recently from 1b087bb to bc6c5ff Compare February 12, 2026 09:15

giuseppe mentioned this pull request Feb 12, 2026

Add an undocumented general "run with RPC service" containers/buildah#6675

Merged

giuseppe force-pushed the splitfdstream branch 7 times, most recently from a0876b6 to d77fbc4 Compare February 17, 2026 12:44

giuseppe force-pushed the splitfdstream branch from d77fbc4 to 6d605ac Compare February 19, 2026 10:12

giuseppe changed the title ~~[WIP] storage: add splitfdstream for efficient layer transfer with reflink support~~ storage: add splitfdstream for efficient layer transfer with reflink support Feb 19, 2026

giuseppe force-pushed the splitfdstream branch from 6d605ac to 0ebb56c Compare February 19, 2026 14:49

giuseppe marked this pull request as ready for review February 19, 2026 14:58

giuseppe force-pushed the splitfdstream branch from 42fcc4e to d9f9694 Compare March 20, 2026 17:57

github-actions bot added common Related to "common" package image Related to "image" package labels Mar 20, 2026

giuseppe force-pushed the splitfdstream branch 2 times, most recently from 88bf912 to a7806a8 Compare March 23, 2026 07:39

cgwalters reviewed Apr 6, 2026

View reviewed changes

giuseppe added 2 commits April 7, 2026 08:53

pkg/fileutils: use CopyFileRange if possible

bf9df7d

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage, chunked: new function GenerateDumpFromTarHeaders

d5428aa

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the splitfdstream branch from a7806a8 to b304a2e Compare April 7, 2026 11:05

giuseppe added 9 commits April 7, 2026 14:19

storage/vendor: add github.com/bootc-dev/jsonrpc-fdpass-go

8141b89

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage/pkg/splitfdstream: new package

360233a

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage/store: add new APIs to resolve image ID

afae4c8

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage: add SplitFDStreamStore interface

a7f4037

Extend the store with splitfdstream capabilities exposed via a UNIX socket for JSON-RPC communication. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage/archive,chrootarchive: add support for splitfdstream

4bce803

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage, overlay: use openat2 instead of using procfs

d2d0b8d

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

composefs: do not leak open file

b7c0055

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

storage/overlay: implement SplitFDStreamDriver

7a56d1d

Implement the SplitFDStreamDriver interface for the overlay driver, enabling efficient layer operations with reflink support. Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

image, storage: expose the underlying SplitFDStreamStore

de76e6f

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the splitfdstream branch 2 times, most recently from 4834b91 to 7af6099 Compare April 7, 2026 13:25

giuseppe force-pushed the splitfdstream branch from 7af6099 to daa8688 Compare April 7, 2026 13:33

		// FDs are streamed to the child process one-at-a-time over a Unix socket
		// using SCM_RIGHTS, avoiding EMFILE from inheriting too many FDs at exec.

		// The library limits to MaxFDsPerMessage per sendmsg, so remaining
		// FDs are sent as follow-up "fds" notifications.

Conversation

giuseppe commented Feb 10, 2026

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

giuseppe commented Feb 10, 2026

Uh oh!

giuseppe commented Feb 10, 2026

Uh oh!

mtrmac commented Feb 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giuseppe commented Feb 11, 2026

Uh oh!

packit-as-a-service bot commented Feb 11, 2026

Uh oh!

TomSweeneyRedHat commented Mar 10, 2026

Uh oh!

mtrmac commented Mar 10, 2026

Uh oh!

giuseppe commented Mar 10, 2026

Uh oh!

TomSweeneyRedHat commented Mar 18, 2026

Uh oh!

giuseppe commented Mar 23, 2026

Uh oh!

giuseppe commented Mar 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cgwalters commented Apr 8, 2026

Uh oh!

giuseppe commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants