UM code-styling script broken?

[MichaelWhitall]

Hi all,

Does anyone know a way to run the UM code-styling script in a git branch?

In UM fcm branches this can still be run as:

module load scitools
./admin/code_styling/apply_styling

But this fails if you run it in a git clone. The UM styling script decides which files to apply to by running fcm commands to see which files your branch has modified. Obviously that doesn't work in a git clone (it just complains mournfully that the current directory doesn't appear to be a valid working copy...)

Any chance the UM styling script can be changed so it works like the lfric_apps one? (where you just specify the directory you want to apply it to, instead if it trying to do something clever to work out which files to re-style).

Cheers!
Mike

[James Bruten (james-bruten-mo)]

Hi Mike,

Yes, sorry, this is something that's been missed in the git migration.
The functionality is still available, but in a slightly different way. The files are in the SimSys_Scripts repo, which has a copy in UMDIR.

# Move to the fixer scripts location
cd $UMDIR/SimSys_Scripts/umdp3_fixer

# Run the rosestem_branch_checker which takes a directory and checks all files within there
./rosestem_branch_checker.py --fixer_source . --source /path/to/um/clone

The rosestem_branch_checker will take some time as it checks all files - unfortunately it was written just for rose-stem and so assumes it's checking from the top of the file tree.
This obviously isn't the cleanest interface, so we'll try and make something a bit smoother over this release.

Cheers

[MichaelWhitall]

Hi thanks for this James,

As far as I can tell though, rosestem_branch_checker.py just checks the code-styling rather than fixing / re-applying it in my branch working copy? (I tried running it in a UM branch into-which I had copied code from lfric_apps, and it issued a fail due to the code being non-compliant but didn't actually fix it?) Is there a way I can run just the "fixer" without it trying to use fcm?

Cheers!
Mike

[James Bruten (james-bruten-mo)]

I thought that file did apply it, but apparently not. In that case, you can use umdp3_fixer.py which is in the same directory as rosestem_branch_checker.py. It takes the paths to files that need fixing as the input.

./umdp3_fixer.py /path/to/file1 [path/to/file2...]

[MichaelWhitall]

That's great thanks James, I'll try that :)

Looking inside rosestem_branch_checker.py, I think what its doing is copying the whole working copy to a tmp dir, applying the fixer to all files in there, then reporting an error if the contents of the tmp dir no-longer match the original working-copy. So you're right it does apply it, but the changes it makes then get thrown out ;)

But running the fixer itself file-by-file sounds good to me (I'm sure I can make a shell script that finds all the .F90 files in a specified directory and runs the fixer on each one...)

Cheers!
Mike

[Jenny Hickson (jennyhickson)]

Hi Mike,

Any chance the UM styling script can be changed so it works like the lfric_apps one?

Are you thinking of SimSys_Scripts/lfric_styling/lfric_styling.py? If so its worth noting that these are quite different scripts. The LFRic one just lower-cases the intrinsic names and is there to aid people copying code from the UM into LFRic. It doesn't apply a full styling.

Cheers

[MichaelWhitall]

Hi thanks Steve,
Yes I've heard it said that the LFRic dynamical core will not bit-reproduce on different decompositions, but there's been some discussion lately about running a physics-only rose-stem test to check bit-comp (i.e. do a full 3D global run, initialised with the usual horizontally-variable weather, and just run it for a day or two with the solver and transport switched off). In theory that should bit-reproduce, since the physics should be independent in each column?

[MichaelWhitall]

Various problems I've stumbled-upon before which show up as failure of the bit-comp tests in the UM:

• Referencing uninitialised values (that should also get snagged by rigorous compile tests, but I have seen cases where the problem was rarely exposed and only showed up in the CCE fast compile runs just because those are run for longer).
• Referencing arrays out-of-bounds and picking up random junk from the neighboring memory-address (again I've seen cases where the problem was rarely-exposed such that the rigorous compile tests didn't hit it, but the longer rose-stem runs did).
• Referencing the wrong grid-point; a common mistake when handling sparse calculations using lists of "compression indices". e.g. if my code makes a list of the i, j indices of points where I need to do calculation A and another list of the subset of points where I need to do calculation B, and I accidentally put the wrong index in, this will create an incorrect translocation of the calculation to a different model-column; that usually shows-up as a "rose_ana_pe" fail in the UM, but there is no such safety-net in LFRic. This is the one I'm most worried about; CoMorph does a lot of sparse calculations on different compression lists and its important to be able to check that the compression indexing has been done right.
• Compiler bugs; sometimes aggressive compiler optimisations can break things. Admittedly, sometimes this breaks reproducibility on different decompositions but is harmless really (e.g. I found a case where, in a vectorised calculation, the compiler was doing something different in the vector loop versus the remainder loop, so that if the processor domain width wasn't a multiple of 4 then you'd get different rounding errors in the remainder grid-cells at the end of the domain). But very occasionally they have genuinely broken things, and failure of the bit-comp test has been the only way I knew about it.

[Steve Mullerworth (stevemullerworth)]

That could be a good test to add. Once it is proven to work, a Rose Stem test comparing different processor configurations could probably be set up.

[MichaelWhitall]

That could be a good test to add. Once it is proven to work, a Rose Stem test comparing different processor configurations could probably be set up.

Awesome :) I reckon it'll be a significant effort to achieve the "Once it is proven to work" step though. Since the LFRic physics has prematurely undergone significant development independent from the UM without any bit-comp test, that test will likely be broken and we'll need to track down the bugs... In the mean-time I'm going to stick with keeping my code interoperable with the UM so I can still use the bit-comp tests there, and I'd encourage others to do the same!

[Steve Mullerworth (stevemullerworth)]

I would be slightly more optimistic about the ability to track down issues as there are a few tools in the bag that can be used to isolate issues, and there is much less use of global data in LFRic which reduces the risk of uninitialised variables. Also, the fast-debug and full-debug tests initialise all fields, operators, and (using compile options) some uninitialised variables with NaN.

If we had a reproducible global sum then we could potentially target the full model because there are ways to constrain order of compute of GungHo kernels.