Installing Flux more easily

[vsoch]

We really need to be able to remove the requirement that flux is installed in the container, either possibly via an app image, layers injected in at runtime, or similar. Do you have any suggestions @ChristianKniep or @pditommaso? Flux has (of course) binaries, config files, and uses system libraries, so it's not so trivial as building one binary and klunking it into a layer and then bin!

[pditommaso]

This is what Wave is made for. You can make an API request and get the for a Wave container image name that represents your source image, plus the layer including Flux.

The only limitation I see is that Wave container image so far are ephemeral, in the meaning it expires after a few hours.

However thinking to your use case, it could be useful, and simple, to add the possibility to have a permanent container name.

[ChristianKniep]

Yup, MetaHub will solve the same issue. I reckon that it's going to be hard otherwise.

On one hand you want to have images that are agnostic to the execution environment and the orchestrator as they are most likely upstream images out of your control. The upstream community does not have an incentive to include all the tools needed downstream (because it would be a stupid idea).

[vsoch]

So MetaHub and Wave are fairly similar it seems? They both take a pull request and modify the manifest and return a different one. It sounds like I should implement something (simple) like this and serve it alongside the MiniCluster? 🤔

[pditommaso]

[vsoch]

Why? Because I need the functionality to add flux to containers at runtime, and I can’t rely on private or paid for services?

[pditommaso]

I'm wrong, or is this project hosted on GitHub, that's a private company of Microsoft? 😅

I don't know what the business model of MetaHub is, but we provide Wave service for free to academic and non-profit orgs.

In any case, I fully understand your point of view.

[vsoch]

Oh I could use Wave? I didn't realize that! I thought it was just integrated into Nextflow / Tower?

[pditommaso]

Yes 👉 #118 (comment)

[vsoch]

okay sweet! I'll try that later today. Thank you @pditommaso and @ChristianKniep !