You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
SPIDER currently uses SciATH as its test framework. SciATH is a custom testing tool developed for PETSc-based applications, but it has several practical problems:
Not pip-installable: SciATH is distributed as a git submodule, not a standard Python package. This makes installation fragile and adds a dependency that is not managed by any package manager.
Not installed on most developer machines: Most PROTEUS developers do not have SciATH set up, so make test fails silently. The validation tests added in PR Add external mesh input and EOS out-of-range safety #5 are standalone Python scripts precisely because SciATH was unavailable.
Inconsistent with the ecosystem: Every other component in the PROTEUS ecosystem (PROTEUS, Zalmoxis, CALLIOPE, JANUS, MORS, ZEPHYRUS, Aragog) uses pytest. Having SPIDER use a different framework creates friction for contributors who work across multiple repos.
Limited CI integration: pytest has mature GitHub Actions integration, coverage reporting, and parallel execution support. SciATH requires custom CI configuration.
No marker system: pytest markers (@pytest.mark.unit, @pytest.mark.smoke, @pytest.mark.slow) allow selective test execution that matches the PROTEUS CI pipeline. SciATH does not have an equivalent.
Proposed approach
Write pytest wrappers for existing SPIDER tests: Each existing SciATH test case becomes a pytest function that runs the SPIDER binary as a subprocess (via subprocess.run) and validates the output. This is the same pattern used in PROTEUS for SPIDER integration tests.
Port the validation scripts from PR Add external mesh input and EOS out-of-range safety #5: The standalone scripts (generate_aw_mesh.py, generate_super_earth_mesh.py, validate_mesh_fields.py) become proper pytest test functions with assertions and markers.
Add standard markers: @pytest.mark.smoke for quick binary-runs-without-crash tests, @pytest.mark.unit for output validation against expected values, @pytest.mark.slow for full-evolution tests.
Remove SciATH submodule: Once all tests are ported, remove the SciATH git submodule and its configuration files.
Add pyproject.toml (or extend existing one) with pytest configuration, matching the PROTEUS ecosystem conventions.
Benefits
Contributors can run pytest in the SPIDER directory and get immediate feedback.
CI can use the same pytest -m smoke pattern as the rest of the ecosystem.
Coverage reporting becomes possible via pytest-cov.
Test discovery and parameterisation work out of the box.
Notes
SPIDER is a C codebase, so the pytest tests would be "integration-style" — they invoke the compiled binary and check output files/exit codes. This is appropriate and matches how PROTEUS already tests SPIDER.
The existing tests/opts/*.opts option files and expected output files can be reused as-is.
Problem
SPIDER currently uses SciATH as its test framework. SciATH is a custom testing tool developed for PETSc-based applications, but it has several practical problems:
make testfails silently. The validation tests added in PR Add external mesh input and EOS out-of-range safety #5 are standalone Python scripts precisely because SciATH was unavailable.@pytest.mark.unit,@pytest.mark.smoke,@pytest.mark.slow) allow selective test execution that matches the PROTEUS CI pipeline. SciATH does not have an equivalent.Proposed approach
Write pytest wrappers for existing SPIDER tests: Each existing SciATH test case becomes a pytest function that runs the SPIDER binary as a subprocess (via
subprocess.run) and validates the output. This is the same pattern used in PROTEUS for SPIDER integration tests.Port the validation scripts from PR Add external mesh input and EOS out-of-range safety #5: The standalone scripts (
generate_aw_mesh.py,generate_super_earth_mesh.py,validate_mesh_fields.py) become proper pytest test functions with assertions and markers.Add standard markers:
@pytest.mark.smokefor quick binary-runs-without-crash tests,@pytest.mark.unitfor output validation against expected values,@pytest.mark.slowfor full-evolution tests.Remove SciATH submodule: Once all tests are ported, remove the SciATH git submodule and its configuration files.
Add
pyproject.toml(or extend existing one) with pytest configuration, matching the PROTEUS ecosystem conventions.Benefits
pytestin the SPIDER directory and get immediate feedback.pytest -m smokepattern as the rest of the ecosystem.pytest-cov.Notes
tests/opts/*.optsoption files and expected output files can be reused as-is.