[MNT] Fix race condition in OpenMLSplit tests during parallel execution #1641

[codervinitjangir]

Metadata

Reference Issue: Fixes #1641

New Tests Added: No (Fixes existing tests)

Documentation Updated: No

Change Log Entry: Fix race condition in OpenMLSplit tests by using unique temporary directories for parallel execution.

Details
What does this PR implement/fix?
This PR fixes a race condition in OpenMLSplitTest that occurs when tests are run in parallel (e.g., using pytest -n). I updated the setUp method to create a unique temporary directory for each test run using tempfile.mkdtemp() and ensured that the input ARFF file is copied into this isolated path. I also added proper cleanup in tearDown using shutil.rmtree().

Why is this change necessary?
Previously, parallel workers were sharing the same file path for the generated .pkl.py3 cache file. This resulted in intermittent EOFError because one worker would delete the file during tearDown while another worker was still trying to read it.

How can I reproduce the issue this PR is solving and its solution?
The issue can be reproduced by running the following command multiple times:
pytest -n 3 tests/test_tasks/test_split.py
Without this fix, the tests fail intermittently (roughly 1 in 10 runs). With this fix, the tests pass consistently in parallel environments.

Any other comments?
Following the maintainer's suggestion in the issue thread, this approach avoids shared paths entirely by giving each worker its own tempfile directory.

[codervinitjangir]

Hi @geetu040 , I have submitted a PR to resolve this race condition using the unique tempfile directory approach we discussed. The CI workflows are currently awaiting approval to run. Could you please trigger them when you have a moment? Thanks!