What's making local builds so slow?

[iboutle]

The local build used to run through very quickly for incremental rebuilds, but now it seems to get stuck for ~10 minutes at the following point:

16:12:48 Building dependencies.mk
16:13:00 Collating programs.mk
16:24:42 ========================================================= 
16:24:42 Compiling lfric_atm 

does anyone know what's going on, and whether it can be sped-up again?

Thanks!

[Steve Mullerworth (stevemullerworth)]

Is this repeatable? Do you build on a local VDI temp persistent disk or elsewhere.

I'm not seeing the same problem (at 3.1) on a temp persistent disk (which is a fast disk). Takes about 30 seconds.

I did see a file system issue on $DATADIR this afternoon about 16.30. File systems other than temp persistent (including $HOME) can be slower for IO access.

[iboutle]

It was repeatable all day yesterday, which is what made me wonder if it was a vn3.1 thing. I haven't noticed it being like this before.

I checkout and build in my $HOME space, which I thought was a fast disk? I thought temp/persistent is specific to each VDI machine, so not really that useful if you get logged in to a different one - is there a fast disk that follows you around the VDI estate?

[Wolfgang Hayek (tinyendian)]

I did notice slow execution of this build step on high-latency network file systems on our HPC as well, while RAM disk execution is much faster. Maybe the ProgramObjects tool performs a lot of small IO queries in the SQLite dependency database?

[Steve Mullerworth (stevemullerworth)]

I believe your analysis is right. Since the build system was written I'm told there have been developments to SQLite that would have enabled this to be done without a lot of small IO accesses, but it's a bit late in the day to implement them!

For the record, the performance at vn3.1 for generating the lfric_atm programs,mk seems about one-third slower than at vn2.2 (~22 seconds up to ~30 seconds on my non-networked disk) which I assume is accounted for by the increase in the number of object files required (though that doesn't seem to have gone up by 1/3).