[uss_qualifier] Add global timeout and run-to-completion requirement#1399
Merged
BenjaminPelletier merged 3 commits intointeruss:mainfrom Mar 25, 2026
Merged
Conversation
the-glu
approved these changes
Mar 24, 2026
Contributor
the-glu
left a comment
the-glu
left a comment
There was a problem hiding this comment.
LGTM, a few nit / questions
| ) | ||
| tested_requirement = _tested_requirement_for(REQ_RUN_TO_COMPLETION, breakdown) | ||
| tested_scenario = _tested_scenario_for(default_scenario, tested_requirement) | ||
| # Assume each TestedScenario for the the TestedRequirement for this requirement should only ever have 1 case with 1 step with 1 check |
Contributor
There was a problem hiding this comment.
nit: Shouldn't the code be defensive against the assertion?
Member
Author
There was a problem hiding this comment.
nit: Shouldn't the code be defensive against the assertion?
Seems like good practice; added (locally; will push shortly).
| @@ -1,11 +1,13 @@ | |||
| from __future__ import annotations | |||
Contributor
There was a problem hiding this comment.
It the statement still needed?
Member
Author
There was a problem hiding this comment.
Yes, interestingly -- both my IDE and make format (via ruff) complain when this is omitted because a class method is returning an instance of that class (from_scenario_report).
barroco
approved these changes
Mar 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
uss_qualifier provides a great deal of useful information in its test report for any given test run. But, that test report information isn't accessible until the test completes. If a test takes far longer than a reasonable amount of time, the information captured so far in the test report would be very useful even if the test never ran to completion. This PR addresses that problem by adding an
stop_afterflag to the execution configuration -- when set to a timedelta, all remaining test actions (scenarios, suites, action generators) will be skipped after elapsed time exceeds this timedelta. So, execution will not stop immediately atstop_after, but it should stop fairly soon after this duration (whenever the currently-running test scenario completes, plus a small amount of time to finalize). This allows a user, for instance, to setstop_afterto 2 hours and then force-stop the process after 2.5 hours to almost always provide access to the information that caused the test run to take more than 2 hours.A concern
stop_afterintroduces is that test behavior becomes somewhat less predictable since process completion doesn't mean the test run actually ran to completion normally. The fact that this had happened could be less obvious than ideal in some cases, so this PR also adds a new specialinteruss.automated_testing.execution.RunToCompletionrequirement. If a user includes this requirement in a tested requirements artifact's requirement set, it will be populated with success upon running to completion normally, or failures providing information about the premature stop if the test run stops early. RunToCompletion failures due to Critical-severity check failure fit nicely into tested requirements' existing layout. RunToCompletion success and RunToCompletion failures due to timeout-based action skips bend the scenario-based breakdown a bit with an N/A scenario for success and (test suite)/(action generator)-prefixed scenario names when those components are skipped. But, RunToCompletion failures due to a test run timeout should be rare, and the content of tested requirements should be of less interest in those cases, so I think this bending is acceptable.The special requirement is added to f3548_self_contained, but would be difficult to continually trigger in our CI as we have view expected-failed test scenarios. I manually verified that the tested requirements artifact looks as expected when a test run ended early due to a Critical-severity check failure, and when a test run ended early due to
stop_after.