Define a set of named features and logic to manage their dependencies.

[albu-diku]

Summary:

This work povides a way to make use of per-feature requirements files while maintaining the ability to have certain requirements overridden during build and other be met by system packages. This would make it suitable for immediate use in docker-migrid where it would replace the series of hand rolled pip install lines gated by feature checks and the package installation lines those checks contain.

Detail:

The codebase has a number of features that, when selected, require certain dependencies. Thus far the handling for this has been split between this repository and the docker-migrid, which encodes a large amount of version information for the various features and in addition allows overriding these at build time.

There is a desire to see these become per-feature requirements files expressed in this repository - that would mean the dependencies are captured alongside the code that depends upon them. However, the ability to override the versions being used and, more critically, to make use of packages provided by the distribution must remain possible.

This PR introduces work that will allow that. The features are explicitly defined by name in a static file that describes them. The added features tool reads this file and, after consulting a source for what is enabled (this can be the environment but also a .env file, as used within the docker builds, directly) will output the lines needed to install the packages.

This simple conceptual design allows us to handle package overrides and distribution packages. We maintain a list of environmental keys to be consulted for certain package versions and if present expand the packages we must install from the requirements file and make use of the an applicable override instead. In the situation of distribution packages, we perform the same requirements expansion but check whether the dependency is already met in the python environment and skip its installation if so.

Review

To anyone taking a look: not all the features are fully defined here as a way to first share the approach. The tests coves all installation cases, but when run directly the repository level notion of defined features are used. Two named features re currently defined, CLOUD and MIGUX (which likely gives you an idea of what I was working on). I would recommend comparing the output from the following commands:

./envhelp/lpython ./mig/install/features.py install --env --check
ENABLE_CLOUD=true ENABLE_WORKFLOWS=true ./envhelp/lpython ./mig/install/features.py install --env --check
ENABLE_CLOUD=true OPENSTACKSDK_VERSION_OVERRIDE=1.2.3 ./envhelp/lpython ./mig/install/features.py install --env --check

[jonasbardino]

I think this is an important and also quite big task we want to pursue so thanks for getting it rolling 👍
How would you feel about adding a new milestone on it and e.g. moving the high-level description and motivation there with this PR then assigned to it?

[jonasbardino]

Being a draft I don't want to go into pedantic mode but just say that it needs an overhaul regarding the usual style fixes including copyright header and docstrings before consideration for merging.
I'll comment on a few other conceptual and overall bits to discuss separately.

[jonasbardino]

Noticing the function to strip versions and considering how we may also want to detect or handle e.g. different features requesting different versions of the same package (say Cryptography) we may want to look into dedicated helpers to reuse. The packaging package has several version and requirements parsing helpers
https://packaging.pypa.io/en/latest/version.html
https://packaging.pypa.io/en/stable/requirements.html
and it is already packaged in RHEL and Debian-based distros. So something to consider to avoid reinventing solid utilities.

[jonasbardino]

I noticed quite a heavy use of f-strings in this PR. We have historically stayed with %-expansion e.g. over f-strings for various reasons, and I think we want to discuss it in the dev team first if that is to change.

[jonasbardino]

I've raised a couple of issues and made a few suggestions, but after looking it over and trying out the command examples you suggested it generally looks good and a step in the right direction.

[albu-diku]

I think this is an important and also quite big task we want to pursue so thanks for getting it rolling 👍 How would you feel about adding a new milestone on it and e.g. moving the high-level description and motivation there with this PR then assigned to it?

That is a great idea and have done just that!

[albu-diku]

Being a draft I don't want to go into pedantic mode but just say that it needs an overhaul regarding the usual style fixes including copyright header and docstrings before consideration for merging. I'll comment on a few other conceptual and overall bits to discuss separately.

Just want to confirm, this was very well known and the review you have given it, at the level of approach, is exactly what was needed to confirm it worth pursuing.

Nonetheless making significant inroads on those notes will be pressing as part of adding the other suggested feature example.

[albu-diku]

I noticed quite a heavy use of f-strings in this PR. We have historically stayed with %-expansion e.g. over f-strings for various reasons, and I think we want to discuss it in the dev team first if that is to change.

Ah yah, actually that switch wasn't entirely conscious beyond realising their use was no longer a worry given our as of recent minimum Python version and then writing this as I would my personal python code.

My own take is f-strings are extremely pleasant for clarity of intent, but I'm certainly not married to this. Not an issue for me to switch it back right away but otherwise perhaps it can serve as a useful case in point for discussion by the team for now.

[albu-diku]

Noticing the function to strip versions and considering how we may also want to detect or handle e.g. different features requesting different versions of the same package (say Cryptography) we may want to look into dedicated helpers to reuse. The packaging package has several version and requirements parsing helpers https://packaging.pypa.io/en/latest/version.html https://packaging.pypa.io/en/stable/requirements.html and it is already packaged in RHEL and Debian-based distros. So something to consider to avoid reinventing solid utilities.

Something perhaps worth a bit of vocal discussion but this was actually rather front of mind for me, hence trying to re-use as much of pip as possible. When I wrote this my thinking was to minimise external dependencies given this is itself a bootstrap tool - but agree it is a very fine line and I think the biggest argument for bringing something in would be one of maintenance i.e. it may well be far better to defer to a formal packaging library (thus requiring it) if it isolates us from the packaging tools themselves moving forward. Definite food for thought.