[WIP] composefs-splitfdstream: new crate by giuseppe · Pull Request #228 · composefs/composefs-rs

giuseppe · 2026-02-10T12:45:02Z

Add a client library for communicating with the containers-storage splitfdstream server. This enables importing OCI container layers via UNIX socket with file descriptor passing.

Needs: containers/container-libs#651

cgwalters · 2026-02-10T13:15:48Z

crates/composefs-splitfdstream/src/protocol.rs

+//! This module implements the JSON-RPC 2.0 protocol types used to communicate
+//! with the containers-storage splitfdstream server. The protocol is line-delimited
+//! JSON over Unix sockets with file descriptor passing via SCM_RIGHTS.


WDYT about https://github.com/cgwalters/jsonrpc-fdpass ? I spent a bit of time on that. I was thinking about transferring it either to github.com/composefs or github.com/bootc-dev.

cgwalters · 2026-02-10T13:18:20Z

crates/composefs-splitfdstream/src/stream.rs

+/// - `i64 LE >= 0`: external FD reference at index `value`
+#[derive(Debug)]
+pub struct SplitFDStreamReader {
+    cursor: Cursor<Vec<u8>>,


why wouldn't this accept any impl Read (or BufRead) per above

cgwalters · 2026-02-10T13:19:21Z

crates/composefs-splitfdstream/src/stream.rs

@@ -0,0 +1,185 @@
+//! Binary stream format parser for splitfdstream.


I did a whole implementation of this over in https://github.com/containers/composefs-rs/pull/218/changes#diff-69fe9e0f1388b28cd8d358d8f76589aaf790016b57fca196c2d22e0f93012b9b - any reason not to unify?

the version here is mostly vibe coded to get it working with the implementation in the storage library.

I guess we can get the other version work too with the c/storage version. Do you mind splitting the splitfdstream implementation from the rest of the PR?

Do you mind splitting the splitfdstream implementation from the rest of the PR?

➡️ #229

cgwalters · 2026-02-10T13:20:04Z

crates/cfsctl/src/main.rs

        digest: String,
+        /// Optional reference name for the layer
        name: Option<String>,
+        /// Import from splitfdstream server socket instead of stdin


This one feels different enough to be a new verb

cgwalters · 2026-02-11T22:15:58Z

crates/composefs-oci/src/splitfdstream.rs

+    ///
+    /// Uses libc recvmsg directly because rustix 1.x has a bug where
+    /// msg_controllen is set to 0, preventing SCM_RIGHTS reception.
+    #[allow(unsafe_code)]


Hmm really? Are you sure? We were successfully using rustix in https://github.com/cgwalters/jsonrpc-fdpass ...

(Also why are we not using that crate? I can publish it formally...ok just transferred it to the bootc-dev org along with the Go version too)

If there's fixes needed in rustix the maintainer is responsive.

We're actually very close to #[forbid(unsafe_code)] in this project, it's just #123

cgwalters · 2026-02-11T22:21:29Z

crates/composefs-oci/src/splitfdstream.rs

+    let mut remaining = size;
+
+    while remaining > 0 {
+        let n = rustix::fs::copy_file_range(


I believe this optimization is already in std https://github.com/rust-lang/rust/blob/7057231bd78d6c7893f905ea1832365d4c5efe17/library/std/src/io/copy.rs#L54 so we can just use std::io::copy

cgwalters · 2026-02-11T22:22:25Z

crates/composefs-oci/src/splitfdstream.rs

+    repo.finalize_object_tmpfile(File::from(tmpfile), size)
+}
+
+fn read_fd_contents(fd: &OwnedFd, size: u64) -> Result<Vec<u8>> {


Why load the whole thing into memory instead of consistently using the O_TMPFILE path?

cgwalters · 2026-02-11T22:23:26Z

crates/composefs-oci/src/splitfdstream.rs

+    }
+    let request = serde_json::json!({
+        "jsonrpc": "2.0",
+        "method": "GetSplitFDStream",


I think this is fine for PoC however bigger picture I think what we need to do is add this method to https://github.com/containers/skopeo/blob/main/cmd/skopeo/proxy.go and then wrap it with https://github.com/bootc-dev/containers-image-proxy-rs/ right?

(And maybe a side quest in there is "port proxy to jsonrpc-fdpass" as new API)

Or in other words... look at what I did in #218 the key thing I think here is fast-pathing the containers-storage: transport using this.

I think in order to do that we really do need to have not just an "import a single layer" as a CLI verb but do the whole "fetch manifest+config + layers" right?

@giuseppe did you see this one? I think to really make this work we need to wire it into the production pull code paths that are used by bootc, not just a lookaside CLI verb right?

Of course doing this is going to lead to conflicts with #218 because it basically wants the same "detect containers-storage, dispatch to different pull path" logic.

My preference honestly is to land that PR first to get us unblocked, then we continue iteration on this one and once we can rely on always having a new enough skopeo/podman we can drop the cstor crate or so?

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

cgwalters · 2026-04-07T13:55:09Z

crates/composefs-oci/src/splitfdstream.rs

+                        let consumed = stream.byte_offset();
+                        self.buffer.drain(..consumed);
+
+                        // Dequeue FDs indicated by the "fds" field.


This code can just use jsonrpc-fdpass rust crate now right?

cgwalters · 2026-04-07T13:59:33Z

crates/composefs-oci/src/splitfdstream.rs

+    }
+    let request = serde_json::json!({
+        "jsonrpc": "2.0",
+        "method": "GetSplitFDStream",


@giuseppe did you see this one? I think to really make this work we need to wire it into the production pull code paths that are used by bootc, not just a lookaside CLI verb right?

Of course doing this is going to lead to conflicts with #218 because it basically wants the same "detect containers-storage, dispatch to different pull path" logic.

My preference honestly is to land that PR first to get us unblocked, then we continue iteration on this one and once we can rely on always having a new enough skopeo/podman we can drop the cstor crate or so?

giuseppe · 2026-04-07T14:13:33Z

@giuseppe did you see this one? I think to really make this work we need to wire it into the production pull code paths that are used by bootc, not just a lookaside CLI verb right?

probably I can even close this PR, I am using it only to test the integration with the container-libs changes.

Of course doing this is going to lead to conflicts with #218 because it basically wants the same "detect containers-storage, dispatch to different pull path" logic.
My preference honestly is to land that PR first to get us unblocked, then we continue iteration on this one and once we can rely on always having a new enough skopeo/podman we can drop the cstor crate or so?

#218 still scares me because it is not clear for how long we'd use that and if something goes wrong with the new API we will be stuck with that implementation, but if you’re confident in the approach, please go ahead.

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe mentioned this pull request Feb 10, 2026

reflink by default (if possible) when copying between containers-storage: instances containers/container-libs#144

Open

giuseppe force-pushed the splitfdstream branch from f59c77c to eb7f1ef Compare February 10, 2026 12:47

cgwalters requested changes Feb 10, 2026

View reviewed changes

giuseppe force-pushed the splitfdstream branch from eb7f1ef to 4f80ec6 Compare February 11, 2026 16:11

cgwalters requested changes Feb 11, 2026

View reviewed changes

giuseppe force-pushed the splitfdstream branch from 4f80ec6 to 9d250d2 Compare February 12, 2026 07:58

cgwalters mentioned this pull request Feb 12, 2026

Add an undocumented general "run with RPC service" containers/buildah#6675

Merged

giuseppe force-pushed the splitfdstream branch 5 times, most recently from c768584 to 1ae8ee8 Compare February 17, 2026 11:59

giuseppe force-pushed the splitfdstream branch 3 times, most recently from d2fd4ef to d00f9ae Compare April 7, 2026 09:15

composefs: new method ensure_object_from_fd

628a4d8

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the splitfdstream branch from d00f9ae to 0ab9182 Compare April 7, 2026 13:31

cgwalters reviewed Apr 7, 2026

View reviewed changes

giuseppe mentioned this pull request Apr 7, 2026

receiver: wait for batched FDs instead of returning MismatchedCount bootc-dev/jsonrpc-fdpass#9

Merged

giuseppe added 2 commits April 8, 2026 08:55

composefs-oci: add support for splitfdstream

f3799e5

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

cfsctl: add import-image-splitfdstream command

63259d0

Assisted-by: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe force-pushed the splitfdstream branch from 0ab9182 to 63259d0 Compare April 8, 2026 06:56

cgwalters mentioned this pull request Apr 8, 2026

storage: add splitfdstream for efficient layer transfer with reflink support containers/container-libs#651

Open

		@@ -0,0 +1,185 @@
		//! Binary stream format parser for splitfdstream.

Conversation

giuseppe commented Feb 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

giuseppe commented Apr 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants