libmultiprocess-ext repo

[ryanofsky]

I think I'd like to create a new bitcoin-core/libmultiprocess-ext repo so we can drop some extraneous things from this repository, particularly documentation, examples, CI scripts, and the shell.nix file, and keep only the core c++ library and tests and CMake files here with a small readme. Advantages:

• This should make Bitcoin Core libmultiprocess subtree smaller and diffs to the subtree easier to review.
• This should make it easier to distinguish PR's which actually affect Bitcoin Core code from PR's that don't and make sure changes affecting Bitcoin Core get the best possible review.
• This should declutter the libmultiprocess repository and increase development velocity. There are a bunch of PRs here like Design Documentation Update #229, parts of Add versions.md and version.h files describing version branches and tags #228, Improve and document act support #225, ci: Replace nix-shell with equivalent nix develop command #223, ci: add newdeps job testing newer versions of cmake and capnproto #212, and parts of ci: Add NetBSD job #196 that I feel like would be better to just merge and iterate on in a unified branch than keep open as PRs that will get stale and do not have great visibility. Since the changes would not affect Bitcoin Core, I don't think they need to be kept open waiting for acks or prolonged review.

If this makes sense, this could be done incrementally. For example CI and documentation could be moved first, since those could be simple moves, while moving examples could require a little more more thought about how we want to structure and reuse the cmake code.

[ryanofsky]

Actually maybe a better alternative to creating a new repo, would just be to create a new empty branch called something like extra in this repo, and open PR's here removing them from the master branch and moving them to extra.

EDIT: Maybe there could be a better name than extra. Chatgpt suggested noncore support infra as alternatives

[maflcko]

How would this deal with ci updates that are conceptually atomic with the code change?

[ryanofsky]

How would this deal with ci updates that are conceptually atomic with the code change?

I can't easily think of examples where this would occur, so wondering if you had a specific case in mind, but I think #228 could dealing with this type of thing straightforward. If you are making an incompatible change you would bump the MP_MAJOR_VERSION number and have the CI script do something different for the new version.

[ryanofsky]

Will keep this issue open for discussion, but am currently thinking having separate repository is actually not a good idea, and just adding a support branch for less essential things would be a better approach, because it would keep discussion and project history unified.

[maflcko]

I can't easily think of examples where this would occur, so wondering if you had a specific case in mind

Looking at https://github.com/bitcoin-core/libmultiprocess/pull/175/files, it seems to modify the cmake file and the ci config, but it should be trivial to split up across the two branches.

[ryanofsky]

Looking at https://github.com/bitcoin-core/libmultiprocess/pull/175/files, it seems to modify the cmake file and the ci config, but it should be trivial to split up across the two branches.

I see. Yeah #175 change should be straightforward to split up.

#196 might be an example that is less straightforward to split up. Like if the netbsd bugfix required modifying cmake or c++ files, they could no longer be modified in the same PR adding the netbsd CI job. The netbsd CI job would either need to be added after the netbsd bugfix, not providing CI coverage in the bugfix PR, or it would need to be added before the bugfix in a disabled state later enabled by a version increase or flag set in the bugfix PR. Either approach would be a little inconvenient, but I think manageable.

Maybe a more general problem is that splitting CI files from code could make unmerged CI and code changes more difficult to test together. I think this would depend on implementation details though so I want to experiment with a new branch and see what this would actually look like.

[Sjors]

I'm not a huge fan of the added (mental) complexity of having two repos.

A better approach might be to move everything Bitcoin Core needs into a subdirectory and then use git subtree split to export it: https://git-memo.readthedocs.io/en/latest/subtree.html

That way, as a developer on this repo I don't have to think about anything, no separate repo or branch. But Bitcoin Core only pulls in a subset of the changes, so review there will be easier.

The only challenge then is to prove that the changes in the subtree branch (e.g. core) match those in the subdirectory on master. But that's easy to automate.

[ryanofsky]

I agree now having two repos would add too much overhead.

But I think having a support branch with documentation, examples, and ci files and a master branch with just the core library and tests would not be too hard to understand or add complexity in most cases. (Cases where someone might have wanted to update main and support files in lockstep should be rare and have many possible workarounds.)

The main reason I'm suggesting this is that I would like the review standard to be higher than it currently is for the core library and tests as IPC becomes less "experimental', and would like the review standard to be lower for documentation and CI changes that can benefit from faster iteration. Having a minimal subtree is more of a nice side-effect.

My current plan is to experiment with github actions in a fork repo, and if I can come up with a something that works well I will open two PR's here: one deleting ci/ doc/ example/ intended to be merged to master and one deleting src/ include/ cmake/ test/ intended to be merged to a support branch.

[Sjors]

I think having a separate support branch would be especially confusing to anyone new to this repository. If it's not the default branch, then the repo looks undocumented and untested. If it is the default branch, now it's difficult to contribute because you need to deal with two branches. Using a subtree split should avoid that issue, because it makes the "core" branch just something the maintainer needs export, not something other developers need to worry about.

[fanquake]

and would like the review standard to be lower for documentation and CI changes that can benefit from faster iteration.

Couldn't this be done without changing anything else? Non core library/test PRs can just be merged with less ACKs.

[Sjors]

That might be fine as well. During a subtree update in Bitcoin Core, the reviewers can be reminded of which dirs are important.

[ryanofsky]

Couldn't this be done without changing anything else? Non core library/test PRs can just be merged with less ACKs.

Yes, that's another option. But it seems not ideal if people reviewing "Update libmultiprocess subtree" PR's in bitcoin core are seeing changes reviewed to different standards if changes that do not affect bitcoin core can just be excluded in a simple way.

Non core library/test PRs can just be merged with less ACKs.

I actually want to let them be merged with no acks. I feel like nix code and CI code and documentation is trapped under a layer of amber and it would be better if these files could treated more like a wiki. I'd still like PR's to be created, so there is a place to discuss all changes, but I think review could be optional. Or at least the review process for nix/ci/documentation filescould be different than the review for build/library/test files with a hard boundary between the two sets of files, whatever the review standards are.

I think having a separate support branch would be especially confusing to anyone new to this repository. If it's not the default branch, then the repo looks undocumented and untested.

I think it should look nearly identical. The master branch should still be the default branch and the main change there would just be git rm -r example/ doc/ ci/ shell.nix. The top-level README.md would still be there, but with URLs updated to point to https://github.com/bitcoin-core/libmultiprocess/blob/support/doc/usage.md instead of https://github.com/bitcoin-core/libmultiprocess/blob/master/doc/usage.md etc. All pull requests and issues would still appear together. Some pull requests would just have support as their base branch instead of master.

I think this could all work well without practical downsides, so I'm going to experiment externally and maybe try to open PR's here implementing this. Obviously I would not want to make any changes unless there is agreement to make them, I just think having two branches with distinct purposes and merge policies could be a promising approach.

[ryanofsky]

The subtree split idea could be another approach too, I need to learn more about that.