Open
Conversation
cds-amal
force-pushed
the
stack/pr4-ui-testing
branch
from
4ee5139 to
d13182a
Compare
Add test-all-nightlies.yml that builds a matrix from the golden file directories under tests/integration/expected/. Each nightly in the matrix runs integration tests and (optionally) UI tests, ensuring that compat code works correctly across the full supported range.
cds-amal
force-pushed
the
stack/pr4-ui-testing
branch
from
d13182a to
eaadea6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We now have per-nightly golden files (PR #144) and per-nightly UI test lists (PR #145), but nothing that actually runs them across the full supported range. If a cfg-gated match arm compiles on the pinned nightly but not on an older one (say, because the breakpoint date is off by a day), we won't find out until someone manually installs that toolchain and builds. That's the gap this PR fills.
A single GitHub Actions workflow (test-all-nightlies.yml) that:
Discovers the nightly matrix dynamically from the golden-file directories under tests/integration/expected/. No hardcoded list in the workflow file; adding a new breakpoint nightly (with its golden files) is sufficient to include it in CI.
For each nightly in the matrix: installs the toolchain, builds the project, runs integration tests (which auto-select the matching golden files), checks out the rust repo at the corresponding commit, and runs UI tests with the per-nightly override lists.
Runs on three triggers: weekly (Sunday 04:00 UTC) to keep the matrix green when the branch is quiet, on push to master, and on manual dispatch for ad-hoc verification after absorbing new breakpoints.
The matrix uses fail-fast: false so that one broken nightly doesn't mask failures in others. Each job is named by its nightly date for easy triage.
Test plan