UM/LFRic Suites Using Git Sources #566
Replies: 6 comments 53 replies
-
|
Hi this sounds awesome thanks so much for working all this out! Just a trivial nit-pick but the last 4 of the links to the example suites above don't actually go to the right suites (a/a/0/0/3 has accidentally copied and pasted d/n/ from the previous link in-place of a/a/, and similar for the others). Anyhow so glad we can science-test code-changes on github now :) Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
Sorry! Fixed now |
Beta Was this translation helpful? Give feedback.
-
|
Hi James. I have been following your instructions in u-aa003/README.md and at the end it says: Using the mirrors requires either setting the For the Met Office what should I be replacing hostname:/path/to/git_mirrors/ with? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Dan, I've messaged you offline |
Beta Was this translation helpful? Give feedback.
-
|
Hi thanks Dan and James; I have exactly the same question; what is path/to/git_mirrors at Met Office? I'm probably not the only one; any chance you can post the answer here for reference? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Mike, Please don't post internal paths on the public discussions. |
Beta Was this translation helpful? Give feedback.
-
|
OK, sorry I didn't know that would be an issue... Are the specific details for Met Office users documented somewhere else internally? There has to be a way for the masses to find out how to use the mirrors without everyone having to email you! Mike |
Beta Was this translation helpful? Give feedback.
-
|
James Bruten (@james-bruten-mo) many thanks for making these available. A few questions about setting them up on Monsoon:
|
Beta Was this translation helpful? Give feedback.
-
|
Ah ok, you've already done the work setting it up! Looking at the suite, I imagine the easiest thing would be to duplicate the |
Beta Was this translation helpful? Give feedback.
-
|
Ah of course! Thanks James Bruten (@james-bruten-mo)! I do have another error now though: INFO:get_git_sources:Modifying Dependencies to use https
Traceback (most recent call last):
File "/home/users/denis.sergeev.ext/cylc-run/u-dr932/run5/bin/merge_sources.py", line 117, in <module>
main()
File "/home/users/denis.sergeev.ext/cylc-run/u-dr932/run5/bin/merge_sources.py", line 81, in main
dependencies = set_https(dependencies)
File "/lustre/ehz2col/collaboration/home/users/denis.sergeev.ext/cylc-run/u-dr932/run5/bin/get_git_sources.py", line 350, in set_https
source = dependencies[dependency]["source"]
TypeError: list indices must be integers or slices, not str
[FAIL] merge_sources.py -d $CYLC_WORKFLOW_RUN_DIR --tokens # return-code=1Could you fix this please? |
Beta Was this translation helpful? Give feedback.
-
|
James Bruten (@james-bruten-mo) prompted us to set up mirror updating on Monsoon3 and this is now done - if you know the location on SPICE, you should be able to guess the username mapping and the location on Monsoon. You could use a localmirrors "keyword" configuration as outlined in https://metoffice.sharepoint.com/sites/ScienceGitMigrationProjectCommsSite/SitePages/Migrating-a-workflow-to-extrac.aspx, adapted for that location. |
Beta Was this translation helpful? Give feedback.
-
|
Ben Fitzpatrick (@benfitzpatrick) thank you very much! It almost worked - but the |
Beta Was this translation helpful? Give feedback.
-
|
Added! Missed that one |
Beta Was this translation helpful? Give feedback.
-
|
Hi what is the correct syntax for pointing to a branch in the dependencies.yaml file? I'm trying to setup a UM SCM workflow at vn14.0 using a git branch for the first time (already got one that uses only the trunk git sources working fine). The change I made to point to my branch instead of main is: This is supposed to compile using my branch from my fork here: But the extract task is failing with: Presumably I need some additional syntax in the line: Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
Yes, this can be skipped for monsoon as the same file system is visible from cylc and hpc nodes. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I'm trying to make use of an lfric_apps branch on Monsoon but having some issues; this thread seemed like the right place to ask. It seems I am able to clone a fork from the mirrors OK, but when I attempt to run the suite it cannot find it. Using Denis Sergeev (@dennissergeev)'s Seems to work fine, but running my suite yields this error: I am using my suite u-dv345 and my dependencies.yaml contains: I have also tried pointing to a local clone by doing: and, following Mohit Dalvi (@mcdalvi)'s suggestion, changing the dependencies.yaml to: But this also fails with the error: Would greatly appreciate some assistance on this. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Alex, I'm not too sure why it's failing there - it works for me. Could you try the following, git clone localmirrors:/MetOffice/lfric_apps.git
cd lfric_apps
git remote add local localmirrors:MetOffice/lfric_apps.git
git fetch local dennissergeev/ice_giants_tfIf that fails, hopefully we get some better output. As for your local source issue, I suspect the way you've checked out the branch, probably means that that ref doesn't exist in it. You should be able to do: git clone https://github.com/dennissergeev/lfric_apps.git
git switch ice_giants_tfand then in your dependencies file, just have: lfric_apps:
- source: git@github.com:MetOffice/lfric_apps.git
ref: 2025.12.1
- source: /path/to/clone/lfric_apps
ref:where the blank ref will take whatever is currently active in that clone |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for this James Bruten (@james-bruten-mo)! Regarding the localmirrors method, doing: git clone localmirrors:/MetOffice/lfric_apps.git
cd lfric_apps
git remote add local localmirrors:MetOffice/lfric_apps.git
git fetch local dennissergeev/ice_giants_tfseems to have fixed the issue, although I am not really sure why... and Denis Sergeev (@dennissergeev) has said to me that he didn't have to do this to get a suite working either. Do you know why this might be? For the local source issue, doing which suggests there might be an issue with the handling of an empty ref in |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, you're right - when merging in a local source, you do need to provide a ref such as the branch name. I'm not too sure why the localmirrors worked there but not in the suite as those are effectively the same commands. |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, how do you include more than one source-code branch from git sources? (so 3 branches if you include the trunk). In my dependencies.yaml file, I tried the obvious: But with this the extract task fails: So I guess its not as simple as you can just add more entries to add more branches? Note I have other workflows which have compiled and run just fine with either one or the other of the two branches (vn14.0_ukca_chemoff_mrphys, vn14.0_comorph_dev), so its not that there's a problem with one of the branches. I also know that these 2 branches do merge cleanly (no files are modified by both of them). So I must just be getting the syntax wrong in dependencies.yaml, and need to do something different if I want 2 branches? Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
Hi Mike, that's the right syntax and it's just worked for me with those branches - https://cylchub/services/cylc-review/view/james.bruten?&suite=um_ex%2Frun14&no_fuzzy_time=0&path=log/job/1/extract/01/job.err |
Beta Was this translation helpful? Give feedback.
-
|
It's failing because it exited in a bad state - it should handle that more nicely though so I'll look to improve it. A clean run is the way forward for now though |
Beta Was this translation helpful? Give feedback.
-
|
OK so I've now tried rerunning my workflow from scratch a couple times and consistently getting the same error as before in the extract task (fails when trying to do Interesting that these branches merge fine in your workflow, so must be a problem with my workflow? I'm trying to run this one: Could it be a software stack thing? Or I need to give the extract task more resources somehow? Cheers! |
Beta Was this translation helpful? Give feedback.
-
|
MichaelWhitall , James Bruten (@james-bruten-mo) was this issue resolved for you? and can 'see' my branch in the local mirror. However, I am still getting this error (exit code 128) for the extract task on the HPC. I am also wondering about this output before the error: which makes it look like only 'git@github.com:MetOffice' is replaced with localmirrors: and not git@github.com:User ? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Mohit, Yes, because the merging the 2nd source requires a commit, you need to have the git email and user set on the HPC. See https://metoffice.github.io/simulation-systems/WorkingPractices/gh_authorisation.html#initial-setup The log message is slightly misleading, we can update that. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks James, |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Dear All,
We have now upgraded our standalone example suites for the UM and LFRic to run using the Github hosted repositories. These are available at:
This changes where the Source code for the models is extracted from, not where the workflows themselves reside. While the workflows can also be moved to github, this may have implications around archiving, and therefore we recommend keeping them in the roses repository for the time being.
Each suite has a README explaining the changes required to use git sources. These are general guidelines which will likely need modifying based on your exact workflow, and the names within.
Defining Dependencies
Sources are defined from a
dependencies.yamlfile in the suite. This format supports multi-branch extraction and automated merging. For example,which will first clone the Met Office lfric_apps repository at tag
2025.12.1and then attempt to merge inmy_branchfrommy_fork. Merge conflicts will be ignored in therose-stemdirectory anddependencies.yaml, otherwise they'll raise an error.Beta Was this translation helpful? Give feedback.
All reactions