Add a variable for explicitly defining pool-size in bake_recipes

[Scott Wales (ScottWales)]

Sites can define environment variable $BUNCH_POOL_SIZE in the bake_recipes task to explicitly set the number of parallel jobs to run rather than relying on the output of nprocs if the correct count is not reported.

Fixes #1988

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

• Documentation has been updated to reflect change.
• New code has tests, and affected old tests have been updated.
• All tests and CI checks pass.
• Ensured the pull request title is descriptive.
• Ensure rose-suite.conf.example has been updated if new diagnostic added.
• Conda lock files have been updated if dependencies have changed.
• Attributed any Generative AI, such as GitHub Copilot, used in this PR.
• Marked the PR as ready to review.

[James Frost (jfrost-mo)]

Looks sensible. I feel we should document this somewhere, but off the top of my head we don't really have anywhere suitable. Maybe we should pass it through the rose-edit GUI, so we can have help text there?

[Scott Wales (ScottWales)]

I don't think the rose metadata is the place to document it. Probably a site would define this in the site file based on the number of CPUs being allocated, e.g.

[[bake_recipes]]
    [[[directives]]]
        -l ncpus = 10
    [[[environment]]]
        BUNCH_POOL_SIZE = 10

Maybe best to describe this in the sample localhost site, or in a README in the site directory explaining how to set up CSET at new sites?

[James Frost (jfrost-mo)]

Hmm, perhaps in the adding a new site documentation then? https://metoffice.github.io/CSET/usage/add-site.html#add-site-file

[James Frost (jfrost-mo)]

I've added a bit of documentation to the add a site page:

If you're happy with this please feel free to merge this pull request.