Add system test coverage for partitioned-heat-conduction + fix issues #404, #674, #402 by RajdeepKushwaha5 · Pull Request #734 · precice/tutorials

RajdeepKushwaha5 · 2026-03-01T21:23:20Z

Summary

This PR contains contributions developed as part of the GSoC 2026 entry test
for the "System Tests Improvements" project.

1. Add `partitioned-heat-conduction` to system tests (entry test)

Adds partitioned-heat-conduction/metadata.yaml defining 2 participants
(Dirichlet, Neumann) and 6 cases across fenics-adapter, nutils-adapter,
and openfoam-adapter.
Adds a new partitioned_heat_conduction_test suite in tests.yaml covering
the three homogeneous solver combinations.
Validated locally with python print_case_combinations.py (output below).
Reference results pending CI run (require Docker/Linux).

print_case_combinations.py output for partitioned-heat-conduction

partitioned-heat-conduction:

(dirichlet-fenics, neumann-fenics)
(dirichlet-fenics, neumann-nutils)
(dirichlet-fenics, neumann-openfoam)
(dirichlet-nutils, neumann-fenics)
(dirichlet-nutils, neumann-nutils)
(dirichlet-nutils, neumann-openfoam)
(dirichlet-openfoam, neumann-fenics)
(dirichlet-openfoam, neumann-nutils)
(dirichlet-openfoam, neumann-openfoam)

2. Fix #404 — user-friendly error when reference results are missing

__unpack_reference_results() now checks for the archive before opening it
and raises a FileNotFoundError with a clear message pointing to
generate_reference_results.py. _run_field_compare() catches it and
returns a clean FieldCompareResult(exit_code=1) instead of a raw exception.

3. Fix #674 — properly escape GitHub step summary markdown

Separates the Markdown summary output from the fixed-width plain-text terminal
output. Adds _escape_markdown_cell() that escapes |, `, *, _,
~ to prevent broken table cells. Success/failure now renders as
:white_check_mark: / :x: icons.

4. Fix #402 — configurable `--timeout` per run

Adds timeout: int = GLOBAL_TIMEOUT field to Systemtest dataclass and a
--timeout CLI argument to systemtests.py. All three
process.communicate() calls (build, run, field-compare) use self.timeout.

Testing

python -c "import ast; ast.parse(open('tools/tests/systemtests/Systemtest.py').read())" — OK
python -c "import ast; ast.parse(open('tools/tests/systemtests.py').read())" — OK
python tools/tests/print_case_combinations.py — partitioned-heat-conduction parsed correctly
Reference result generation — requires Docker/Linux CI

Registers the partitioned-heat-conduction tutorial with the system test framework. Defines two participants (Dirichlet, Neumann) and six cases across three solver adapters: fenics-adapter, nutils-adapter, and openfoam-adapter. Also adds the reference-results/ directory placeholder and a corresponding partitioned_heat_conduction_test suite in tests.yaml covering the three homogeneous solver combinations (FEniCS, Nutils, OpenFOAM). Related to the GSoC 2026 entry test requirement (add a tutorial to the system tests that does not yet have a metadata.yaml).

…ing (precice#404) Previously, a missing reference archive caused a raw Python exception (tarfile.TarError or FileNotFoundError) that was difficult to interpret. Now __unpack_reference_results() detects the missing file before attempting to open it and raises a FileNotFoundError with a clear message pointing the user to generate_reference_results.py. _run_field_compare() catches that error and returns a FieldCompareResult(exit_code=1) with the message in stderr_data so it surfaces cleanly in the final table.

…limit (precice#402) The simulation timeout was previously hardcoded as GLOBAL_TIMEOUT = 900 s in Systemtest.py with no way to change it at runtime. Changes: - Systemtest dataclass gains a 'timeout: int = GLOBAL_TIMEOUT' field; all three process.communicate() calls (build, run, field-compare) now use self.timeout instead of the module-level constant. - systemtests.py gains a --timeout CLI argument (default 900 s) whose value is forwarded to every Systemtest instance. - Also separates the GitHub step summary markdown rendering from the plain-text terminal output and properly escapes pipe characters and other Markdown-special symbols in cell content (fixes precice#674). Usage examples: python systemtests.py --suites=fenics_test --timeout=300 # catch hangs fast python systemtests.py --suites=fenics_test --timeout=3600 # slow machines

Copilot

Pull request overview

Adds system-test scaffolding and runner improvements to support running the partitioned-heat-conduction tutorial in the system-tests framework, while improving robustness and CI summary rendering.

Changes:

Register partitioned-heat-conduction in the system test metadata and add a new partitioned_heat_conduction_test suite with homogeneous solver combinations.
Improve system test reporting by separating terminal output from GitHub Step Summary Markdown and escaping table cell content.
Improve system test robustness via configurable per-run --timeout and a clearer error path when reference results are missing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`tools/tests/tests.yaml`	Adds `partitioned_heat_conduction_test` suite and references expected reference-result archives.
`tools/tests/systemtests/Systemtest.py`	Adds Markdown escaping for GHA summary, handles missing reference results, introduces per-test timeout usage.
`tools/tests/systemtests.py`	Adds `--timeout` CLI flag and passes it into `Systemtest`.
`partitioned-heat-conduction/reference-results/.gitkeep`	Adds instructions placeholder for reference results location/generation.
`partitioned-heat-conduction/metadata.yaml`	Adds system-test metadata describing participants and cases for the tutorial.

Comments suppressed due to low confidence (1)

tools/tests/systemtests/Systemtest.py:503

Timeout/error handling in _build_docker(): in the exception path you call process.communicate(timeout=SHORT_TIMEOUT) before killing the process. If the initial failure was a TimeoutExpired/hang, this can raise again and leave the docker compose process running. Handle subprocess.TimeoutExpired explicitly and ensure the process is killed (ideally the whole process group) in a finally block before attempting a short communicate/drain.

            try:
                stdout, stderr = process.communicate(timeout=self.timeout)
            except KeyboardInterrupt as k:
                process.kill()
                # process.send_signal(9)
                raise KeyboardInterrupt from k
            except Exception as e:
                logging.critical(
                    f"systemtest {self} had serious issues building the docker images via the `docker compose build` command. About to kill the docker compose command. Please check the logs! {e}")
                process.communicate(timeout=SHORT_TIMEOUT)
                process.kill()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-01T21:28:29Z

tools/tests/systemtests/Systemtest.py

            try:
-                stdout, stderr = process.communicate(timeout=GLOBAL_TIMEOUT)
+                stdout, stderr = process.communicate(timeout=self.timeout)
            except KeyboardInterrupt as k:
                process.kill()
                raise KeyboardInterrupt from k


The exception handling around this docker compose execution appears broken because the outer error path later in this method calls logging.CRITICAL(...) (CRITICAL is an int constant, not a logger function). That would raise a TypeError and hide the real failure. Please change it to logging.critical(...) (or logging.exception(...)) so errors are reported correctly.

Copilot · 2026-03-01T21:28:29Z

tools/tests/systemtests/Systemtest.py

 def display_systemtestresults_as_table(results: List[SystemtestResult]):
    """
-    Prints the result in a nice tabluated way to get an easy overview
+    Prints the result in a nice tabluated way to get an easy overview.


Docstring typo: "tabluated" should be "tabulated".

Suggested change

Prints the result in a nice tabluated way to get an easy overview.

Prints the result in a nice tabulated way to get an easy overview.

Copilot · 2026-03-01T21:28:30Z

tools/tests/systemtests/Systemtest.py

        with tarfile.open(self.reference_result.path) as reference_results_tared:
            # specify which folder to extract to
            reference_results_tared.extractall(self.system_test_dir / PRECICE_REL_REFERENCE_DIR)


Using tarfile.extractall() on an archive without validating member paths can allow path traversal (e.g., entries with "../") and overwrite files outside the target directory. Please implement a safe extraction that rejects absolute paths and parent-directory traversals before extracting.

Copilot · 2026-03-01T21:28:30Z

tools/tests/systemtests.py

+    parser.add_argument(
+        '--timeout',
+        type=int,
+        default=GLOBAL_TIMEOUT,
+        help=(
+            f'Maximum number of seconds to wait for each docker-compose process '
+            f'(build, run, or field-compare) before killing it and marking the '
+            f'test as failed. Defaults to {GLOBAL_TIMEOUT} seconds. '
+            f'Increase this value for slow machines or large simulations; '
+            f'decrease it to catch hanging tests faster.'
+        )
+    )


The new --timeout flag controls the docker-compose process timeout, but issue #402 requests overriding preCICE's in precice-config.xml for selected tests. If this PR intends to close #402, it likely needs additional YAML plumbing + XML modification (or the PR title/issue linkage should be updated to avoid implying #402 is fixed).

Copilot · 2026-03-01T21:28:30Z

tools/tests/tests.yaml

+  partitioned_heat_conduction_test:
+    tutorials:
+      - path: partitioned-heat-conduction
+        case_combination:
+          - dirichlet-fenics
+          - neumann-fenics
+        reference_result: ./partitioned-heat-conduction/reference-results/dirichlet-fenics_neumann-fenics.tar.gz
+      - path: partitioned-heat-conduction
+        case_combination:
+          - dirichlet-nutils
+          - neumann-nutils
+        reference_result: ./partitioned-heat-conduction/reference-results/dirichlet-nutils_neumann-nutils.tar.gz
+      - path: partitioned-heat-conduction
+        case_combination:
+          - dirichlet-openfoam
+          - neumann-openfoam
+        reference_result: ./partitioned-heat-conduction/reference-results/dirichlet-openfoam_neumann-openfoam.tar.gz


This new test suite references reference_result archives that are not present in the repository (partitioned-heat-conduction/reference-results currently only contains .gitkeep). As-is, running this suite will always fail until the corresponding .tar.gz files (likely Git LFS pointers) are added, or the suite is gated/removed until reference results land.

Suggested change

partitioned_heat_conduction_test:

tutorials:

- path: partitioned-heat-conduction

case_combination:

- dirichlet-fenics

- neumann-fenics

reference_result: ./partitioned-heat-conduction/reference-results/dirichlet-fenics_neumann-fenics.tar.gz

- path: partitioned-heat-conduction

case_combination:

- dirichlet-nutils

- neumann-nutils

reference_result: ./partitioned-heat-conduction/reference-results/dirichlet-nutils_neumann-nutils.tar.gz

- path: partitioned-heat-conduction

case_combination:

- dirichlet-openfoam

- neumann-openfoam

reference_result: ./partitioned-heat-conduction/reference-results/dirichlet-openfoam_neumann-openfoam.tar.gz

Copilot · 2026-03-01T21:28:30Z

partitioned-heat-conduction/reference-results/.gitkeep

@@ -0,0 +1,11 @@
+# Reference results for partitioned-heat-conduction are stored as Git LFS archives.
+#
+# To generate them locally, run from tutorials/tools/tests:


The instructions say to run from "tutorials/tools/tests", but in this repository the path is just "tools/tests" from the repo root. Please update this path to avoid confusing users.

Suggested change

# To generate them locally, run from tutorials/tools/tests:

# To generate them locally, run from tools/tests:

RajdeepKushwaha5 added 3 commits March 2, 2026 02:09

Copilot AI review requested due to automatic review settings March 1, 2026 21:23

Copilot started reviewing on behalf of RajdeepKushwaha5 March 1, 2026 21:23 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

PranjalManhgaye mentioned this pull request Mar 2, 2026

feat(systemtests): add per-test max_time override for precice-config closes#402 #738

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add system test coverage for partitioned-heat-conduction + fix issues #404, #674, #402#734

Add system test coverage for partitioned-heat-conduction + fix issues #404, #674, #402#734
RajdeepKushwaha5 wants to merge 3 commits intoprecice:developfrom
RajdeepKushwaha5:gsoc2026/system-tests-improvements

RajdeepKushwaha5 commented Mar 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 1, 2026

Uh oh!

Copilot AI Mar 1, 2026

Uh oh!

Copilot AI Mar 1, 2026

Uh oh!

Copilot AI Mar 1, 2026

Uh oh!

Copilot AI Mar 1, 2026

Uh oh!

Copilot AI Mar 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	Prints the result in a nice tabluated way to get an easy overview.
	Prints the result in a nice tabulated way to get an easy overview.

	# To generate them locally, run from tutorials/tools/tests:
	# To generate them locally, run from tools/tests:

Uh oh!

Conversation

RajdeepKushwaha5 commented Mar 1, 2026

Summary

1. Add partitioned-heat-conduction to system tests (entry test)

2. Fix #404 — user-friendly error when reference results are missing

3. Fix #674 — properly escape GitHub step summary markdown

4. Fix #402 — configurable --timeout per run

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

1. Add `partitioned-heat-conduction` to system tests (entry test)

4. Fix #402 — configurable `--timeout` per run