MPI multiple Threads & Processes behaviour

[qbacpey]

Different MPI implementation might yield differnet Threads & Processes behaviour. For example, in Ubuntu 24.04.1 LTS (WSL), with Open MPI: 4.1.6. Even in one rank, there would be multiple Processes and multiple Threads when the application is running with mpirun. Below are the function tracing of a one rank application.

$ mpicxx test_aio_suspend.cxx -o test_aio_suspend && LD_PRELOAD=libtmio.so mpirun -np 1 ./test_aio_suspend > temp_disable_bf_main.ansi
$ grep -oP '\[pid \d+ \| tid \d+\]' temp_disable_bf_main.ansi | sort | uniq -c
     38 [pid 364928 | tid 136393363814080]
   3251 [pid 364928 | tid 136393412193920]
     51 [pid 364931 | tid 129863704782528]
      2 [pid 364931 | tid 129863785129664]
   1241 [pid 364931 | tid 129863793525760]
      1 [pid 364931 | tid 136393412193920]

As for simplicity, TMIO has these assumptions & tools:

• IO tracing: TMIO would only trace the IO request fired by main thread in main process (i.e. the one call MPI_Init and MPI_Finalized), by using static thread_local to the tracing instance..
• Function tracing: Use the prepossor flag FUNCTION_INFO to control.
  • If you want to see the full tracing, including the function tracing from non-main thread & process, use FUNCTION_INFO == 1.
  • If you only want to see the tracing from main process, or want to classify the tracing base on Ranking, use FUNCTION_INFO == 2.
  • If you only want to see the tracing from main thread, use FUNCTION_INFO == 3.