How to adapt the XManager API for HPC

[ethanluoyc]

Hi,

Thanks for sharing this repo. I really like the launch API design.

I am interested in implementing an implementation module that would allow me to launch experiments on an HPC cluster (e.g., Slurm, Grid Engine, etc.). During prototyping, I found that I was unsure how some concepts in an HPC cluster would fit into the core XManager API.

In particular, one of the concepts in many HPC clusters is that of an array job, which is to launch an identical executable with a different parameter configuration to do, for example, a hyperparameter sweep. This is the recommended way of launching many jobs on the HPC cluster to reduce scheduling overhead. This means creating many work units with identical job descriptions and resource requirements and batching and creating a single job on the HPC cluster.

Looking at the xm_local implementation, I noticed it is slightly tricky to adapt that to work in this setting as the implementation launches each work unit separately. I tried a few approaches to change the API to be able to do array job submission but I am still not satisfied with the approach I have. I was wondering if the developers have some recommended ways of implementing array-job-like behavior on top of the core launch API?

Many thanks in advance.