Where can I (should I) add a temp logical in LFRic?

[MichaelWhitall]

Hi all,

I have a minor bug-fix that I want to lodge in lfric_apps, affecting the PC2 cloud-scheme (the final item in the bullet-list on this issue page: MetOffice/lfric_apps#243). I think in the past it would've been said, if it clearly fixes a bug, just change the code without using any switches as noone will mind a KGO-change for that. But as we get ever-closer to having a frozen and highly-tuned LFRic GC6, I'm guessing that's no-longer the case? In the UM I always ended up having to lodge cloud-scheme bug-fixes under "temporary logicals", because they usually cause a small change in cloud radiative forcing (which is something that has to be finely tuned in the climate model and so mustn't be changed in a frozen configuration).

What's the process for lodging this sort of bug-fix in lfric_apps? I've found the old UM science_fixes_mod still exists in lfric_apps, but could be retired as currently um_physics_init just sets the relevant logicals to true; none of them are in a namelist. Should I just add a switch to toggle on/off the bug in the LFRic cloud namelist? Or does lfric_apps have its own equivalent to temp logicals somewhere that I've missed?

Cheers!
Mike

[iboutle]

I'm open to opinions here, but my current suggestion would be to put it in the cloud namelist.

My reasoning is that whilst the temp_fixes namelist in the UM seemed like a good idea at the time, it's now massive. Personally, I find it very confusing to look at, because it's just a big bucket of fixes to lots of different schemes - there's no grouping or ordering of them, or way of establishing whether a particular fix is relevant to your particular configuration.

Putting it in the appropriate science namelist to which the bugfix belongs makes it clear that this is a bugfix to the cloud scheme - it gives it some clear ownership, and enables it to easily be triggered on/off depending on other options that determine whether the fix is actually relevant or used (i.e. in this case whether PC2 is on or off).

[MichaelWhitall]

Awesome thanks, I was thinking similar :) I certainly wouldn't want to bring back the horrors of the UM trac "temp logicals summary" table (which effectively forbid you from lodging a bug-fix until every last config-owner had signed-off whether they want the fix on or off in their rose-stem test). I'm more than happy to just put this in the cloud namelist and not go back to that!

[Steve Mullerworth (stevemullerworth)]

Will leaving it in the cloud namelist result in a higher likelihood of retiring the bug? I guess the aspiration of the temp_fixes namelist was that it would encourage work to retire them by keeping them in a dedicated group.

[iboutle]

That was definitely the aspiration, although it never really materialised! I'm (perhaps naively) thinking that code owners might be more keen to retire unnecessary code from things they have definitive ownership of? I think the main issue is really the timescale of it all, i.e. a temp-fix added now has really missed the boat for GC6, hence won't get adopted until GC7. Hence we won't be able to retire it until there are no operational configurations still running GC6, which is almost certainly going to be some time in the 2030s!

[R Sharp (r-sharp)]

or way of establishing whether a particular fix is relevant to your particular configuration.

If you don't know - it isn't.
If it's relevant to your configuration, it was set to be 'active' in the configuration you inherited when setting your job up. No-one writes a config from scratch.
no-one goes through old configs to trim out the bits that should no longer be there unless they've got an issue and tracking it back takes them to a config setting.

That's why we collected them all into one place, where they could be seen by folk whose job it was to care about the state of the code over their next publication.

[Jenny Hickson (jennyhickson)]

I've created an issue #604 to remind us to update the working practices to reflect the UM and LFRic having different systems.

I agree that the good intentions behind the UM way have been hard to enforce and implement around this. I'm not sure that having the variables in the namelists for the schemes will make us any more or less likely to retire them - but at least they will be more visible to the scheme owners and I think that is a plus.

I think we still want a way to identify these fixes as being different from the other variables in the scheme rather than just a free-for-all. Perhaps a naming scheme that would identify them and make them grep-able?

[R Sharp (r-sharp)]

Steve's guess aligns with my recollection. The intention was that the temp logicals were 'retired' within a year.

The thorn in the side being it's not up to code owners to decide to retire them. In fact most code owners tend to run the latest version of their code and so once the bug is fixed, that's the version they run.

It's config owners who have top do all the testing and jump thourgh the hoops to determine that the 'fix' doesn't negatively effect the output in a way that can't subsequently be tuned out, and do the long hard work of that tuning. Config owners have no real motivation to do this because it's a lot of work, to get an answer as similar as possible to the one they've already got.

temp logicals was an attempt to bring a 'stick' to the discussion to providfe motivation, by making it easier for the technical maintainers (i.e. us) to see the tech debt and nag the config owners into action.
Either it wasn't a big enough stick, or we're (collectively) too nice, because as can clearly be seen, it didn't work.

I think that in order to get rid of such switches, you have to bite the bullet and say they get physically retired (the code is removed) when the next op model goes to "freeze" - no exceptions. i.e. if you still want to run an old scientific model on a newer release of the code, you either have to backport the 'bug' in a branch in your config or re-tune to as close as possible to get the desired answer.
This would put the onus to do the work on the users who want to see the effect (i.e. reep the benefit or endure the cost). However, as commented above, I think we're (collectively) too nice and would not be determined enough to take that hard a stance.

I would be slightly against loosing the switches in the scheme namelists as that's ju7st sweeping them under the carpet. But I can't see a viable, acceptable alternative to achieve the aim temmp logicals set out to achieve.

[R Sharp (r-sharp)]

As for :

Putting it in the appropriate science namelist to which the bugfix belongs makes it clear that this is a bugfix to the cloud scheme - it gives it some clear ownership, and enables it to easily be triggered on/off depending on other options that determine whether the fix is actually relevant or used (i.e. in this case whether PC2 is on or off).

Thats the approach that existed before we instigated temp logicals. It worked even less well, which is why temp logicals was instigated.

[iboutle]

Yes, I agree we still do want some way of tagging and identifying them...

If we were to put them in the science namelists, then perhaps a naming convention something like "temp_fix_xxx" ? That should make them distinct, easily identifiable and searchable.

Alternatively, if we were to keep the temp fixes namelist, then equally I think we should have a naming convention something like "cloud_xxx", i.e. something that identifies clearly the scheme to which the fix belongs - this is my main bugbear with the UM system, that it isn't quickly obvious from looking at the namelist who Roddy's bit stick should be aimed at.

Either having them in a scheme namelist, named "temp_fix", or having a temp_fix namelist with them named by scheme name satisfies both - it groups them together and clearly identifies their owner.

The other key requirement that the UM doesn't adhere to is the metadata triggering being correct - there are items in the temp fixes namelist which don't get trigger ignored when the scheme or feature they pertain to is turned off. I think it's easier to get this triggering right if the switch is in the same namelist as the thing that triggers it, but in principle it should work either way - it just needs to be done.

[R Sharp (r-sharp)]

Andrew Coughtrie (@andrewcoughtrie) : " Would it be possible to have the number be to an issue that is targeted at removing the temp logical? that way we would could have them all listed in a GitHub project board to track what we have."
That is however essentially a modernised, slightly sleeker version of what we have tried in the past.
Every time a new temporarly logical was added, it was (manually) added to a trac page listing the ticket that added it, the date, and the theoretical removal date.
Opening a ticket/PR to say "we plan to remove this" and then shoving it off into the dark recesses of "no-one has time for this" doesn't really add motivation to tackle the problem.

The recording part is (relatively) easy, what's needed is a way of increasing the motivation to act. Which is why I think making the switches harder to turn on, and tied to the specific config they were used to maintain compatability of results with is worth a try.

As I said before, sadly the folk we need to asist and motivate are the creators of the new configs being set up. We need them to turn those switches off and re-tune the config as a result. Only when the switch is no longer in active use can we, or the code owners, tidy up and remove the switch. The people who get to decide when that get's done, are not (curerntly) those that want to do it, but are the one's with the most work to do and possibly the least benefit to gain. (unless we can come up[ with an "auto-tuner" to adjust variou8s parameters to get a scientifically acceptable/similar answer when those switches are turned off - that would be worth it's coders weight in gold if it worked, maybe it's an area AI/ML could have a stab at, but not knowing a great deal about how to train such things, it may also involve a prohibitive amount of effort to get off the ground)

[R Sharp (r-sharp)]

MichaelWhitall : " So the system for retiring temp logicals ought to be tied directly to the decision about when a given GA / RA configuration is no longer being supported."

Alas the problem is, the switch is left on in the next GA / RA config, and the next and the one after that and so it's always in use by a config which is considered "supported". It's a serious amount of effort to test and retest the cumulative effect of that switch being off in a long production run and what parameters need adjusting to keep the evolution within the desired range during runs agains potentially centuries of obs and results. But it's really easy to leave the switch 'on' as we already know that's been proven to give believable answers in those tests.

[Steve Mullerworth (stevemullerworth)]

r-sharp said: "Opening a ticket/PR to say "we plan to remove this" and then shoving it off into the dark recesses of "no-one has time for this" doesn't really add motivation to tackle the problem."

My suggestiong included the idea of creating a Github project that included all the issues. The project can be routinely monitored by the LFRic apps governance group which ought to include the main configuration owners. Ideally, concerted efforts should be made to agree dates for retirement as and when relevant configurations reach end of life.

By the time GC6 reaches retirement, we could probably have a GitHub AI agent to strip out the relevant code...

[R Sharp (r-sharp)]

"By the time GC6 reaches retirement, we could probably have a GitHub AI agent to strip out the relevant code..." 🙏

By the time GC6 reaches retirement, so will half the folk reading this at time of posting...

[MichaelWhitall]

MichaelWhitall : " So the system for retiring temp logicals ought to be tied directly to the decision about when a given GA / RA configuration is no longer being supported."

Alas the problem is, the switch is left on in the next GA / RA config, and the next and the one after that and so it's always in use by a config which is considered "supported". It's a serious amount of effort to test and retest the cumulative effect of that switch being off in a long production run and what parameters need adjusting to keep the evolution within the desired range during runs agains potentially centuries of obs and results. But it's really easy to leave the switch 'on' as we already know that's been proven to give believable answers in those tests.

Is this so? I distinctly remember Martin Willet switched all the existing temp logicals to use all the bug-fixes lodged at the time when putting together GC5. Obviously more temp logicals got lodged since then, but again all existing fixes have been switched on for GC6. There's always a need for significant retuning when putting together a new GC or RA anyway, so incorporating bug-fixes into a new config is not extra work and GMED / RMED have done it well in recent years.

However there are a bunch of rose-stem tests that exist to test a specific piece of functionality and don't correspond to any particular GC or RA config; I think those tend to be the ones that end up with "out of date" settings in them. But just switching on bug-fixes in those shouldn't be a problem. This would sensibly be done at the same time as when the GC / RA config that still had the bug-fix off gets retired?

Cheers!
Mike

[Steve Mullerworth (stevemullerworth)]

I just talked to Yash and we agreed it could be something that is discussed at the next LFRic Apps Governance meeting.

While that doesn't answer the question above, Yash thought the idea of managing it with a GitHub project (kanban) board was reasonable and should ensure the temporary logicals are not forgotten whatever they are named. A meaningful name, perhaps with reference to the PR in the metadata and code could be useful in identifying items to people reading the code.