Upgrade macro command "self.remove_setting" intermittently not working?

[MichaelWhitall]

Hi all,

In my PR: MetOffice/lfric_apps#406
Among other things, I've attempted to tidy-up some variable-names in the "cloud" namelist, to name the multi-option switches consistently (e.g. dropping the preceding i_ in the variable-names, which apparently is a UM-ism that's now frowned-upon!)

To change the name of a namelist variable, one simply needs to make an upgrade macro which reads the existing value, deletes the existing variable, and adds a new one with the new name but the same value as before. I've done this in versions.py‎, from L64.

When I run the apply_macros.py script with this macro and the consistent branch meta-data, all the app rose-app.conf files seem to have the new variable names added, but in a seemingly random selection of them the old variable names don't seem to be deleted. See the commit from running apply_macros.py in my test branch: MetOffice/lfric_apps@011f872

e.g. the file: ‎rose-stem/app/gravity_wave/rose-app.conf has fully been modified as expected including the deletion of the old variable-names i_bm_ez_opt, i_pc2_erosion_numerics, pc2ini and the adding of the same settings with the new names bm_ez_opt, pc2_erosion_numerics, pc2_init_method. But then if I look at rose-stem/app/jedi_forecast_pseudo/rose-app.conf, the new names have been added, but the old ones have not been deleted. Its as if the macro commands self.remove_setting(...) have randomly worked fine for some apps, but quietly done nothing in others.

Any ideas what's going on / how I can fix this?

Cheers!
Mike

[MichaelWhitall]

OK so I thought I'd investigate by trying to upgrade one of the apps where the self.remove_setting had failed "by-hand" (using rose app-upgrade). In my test branch (where I'd already run the apply_macros.py script to copy the branch macro to wherever its needed), I first reverted one of the problem apps back to how it was before I ran apply_macros.py (by copying the file from my dev branch):

git checkout pc2_homog_reversible -- rose-stem/app/adjoint_tests/rose-app.conf

Then I attempted to upgrade it "by-hand":

cd rose-stem/app/adjoint_tests
rose app-upgrade -M ../../../rose-meta -a vn3.1_t247

This fails, with the following output:

[U] Upgrade_vn3.0_t108-vn3.1_t247: changes: 9
    namelist:linear_physics=Blevs_m=15
        Added with value '15'
    namelist:cloud=pc2_homog_g_method='cf'
        Added with value "'cf'"
    namelist:cloud=i_bm_ez_opt='subcrit'
        Removed
    namelist:cloud=i_pc2_erosion_numerics='implicit'
        Removed
    namelist:cloud=pc2ini='bimodal'
        Removed
    namelist:cloud=bm_ez_opt='subcrit'
        Added with value "'subcrit'"
    namelist:cloud=pc2_erosion_numerics='implicit'
        Added with value "'implicit'"
    namelist:cloud=pc2_init_method='bimodal'
        Added with value "'bimodal'"
    =meta=lfric-adjoint_tests/vn3.1_t247
        Upgraded from vn3.0_t108 to vn3.1_t247
Accept y/n (default n)? y
[FAIL] Error: case mismatch; 
[FAIL]  Blevs_m does not match blevs_m, please only use lowercase.
[FAIL] 

What's baffling is that the error is to-do with the setting linear_physics=blevs_m, which my branch upgrade macro doesn't go anywhere near. And it says its upgrading from vn3.0_t108, but this is a branch from stable at vn3.1 so all the apps should be at vn3.1 before I run my macro.

I think what's happened here is that the last macro in rose-meta/lfric-adjoint_tests/version30_31.py hasn't been applied to the trunk rose-stem apps because its broken on the trunk? Its trying to add a setting called Blevs_m, but the corresponding rose meta-data names this setting blevs_m, so the macro on the trunk fails due to case-mismatch of the letter B.

Is this a known issue on the trunk, and I need to upgrade to a slightly later revision of stable to pick up the fix? But this still doesn't explain why running the apply_macros.py script, it all seems to work OK except that self.remove_setting fails to work in certain apps!

[MichaelWhitall]

Ah just found the earlier post: #596
So this is indeed a known issue. I'll try upgrading to vn3.1.1...
(still baffled as to why the problem manifests as just a few of the macro commands silently not working with no reported errors when I run apply_macros.py!)

[James Bruten (james-bruten-mo)]

Hi Mike,

Yep, you're right, this should be solved if you merge in the current stable branch. I'm guessing you're just a bit unlucky with when you created or last updated your development branch.
The issue is that rose app-upgrade doesn't raise an error code when it reaches the please only use lowercase error, so it wasn't spotted. Hopefully this is getting modified.

Cheers