Skip to content

Add --skip flag to exclude tests by name pattern#660

Open
OmChillure wants to merge 2 commits intokarva-dev:mainfrom
OmChillure:add-exclude-tests-with-skip-patterns
Open

Add --skip flag to exclude tests by name pattern#660
OmChillure wants to merge 2 commits intokarva-dev:mainfrom
OmChillure:add-exclude-tests-with-skip-patterns

Conversation

@OmChillure
Copy link
Copy Markdown

@OmChillure OmChillure commented Apr 10, 2026

Closes #549.

This adds a --skip flag to karva test that excludes tests whose fully qualified name matches a regular expression the inverse of --match. Multiple patterns can be combined; a test is skipped if it matches any of them.

- karva test --skip slow_
- karva test --skip "test_integration_"
- karva test --skip slow_ --skip integration_   # skips either
- karva test -m auth --skip auth_legacy         # match auth, then drop legacy

Implementation

The flag reuses the existing NameFilterSet type from karva_metadata — no new filter type was needed. The skip logic sits alongside the name-filter check in should_skip_variant: a test is filtered out if name_filter rejects it or skip_filter matches it, computing display_name only once when either filter is active.

Invalid regex patterns are caught upfront in the main process (before any workers spawn), consistent with the existing --match and --tag validation, so users get a clean error with exit_code: 2 immediately.

The flag is forwarded from the orchestrator to each worker via --skip in inner_cli_args, mirroring how --match is forwarded.

Changes

  • karva_cli --skip <SKIP_PATTERNS> field on SubTestCommand
  • karva_metadata — skip_filter: NameFilterSet on TestSettings with setter on ProjectSettings; default in TestOptions::to_settings
  • karva/src/commands/test/mod.rs : upfront regex validation
  • karva_runner/src/orchestration.rs : forwards --skip to worker args
  • karva_worker/src/cli.rs : parses patterns and applies set_skip_filter
  • karva_test_semantic : filter applied in should_skip_variant
  • Integration tests in crates/karva/tests/it/skip_filter.rs covering: substring match, multiple patterns (OR semantics), no matches, invalid regex error, and combined with --match

Screenshots

A] All tests running
Screenshot from 2026-04-10 23-54-19

B] Skipping tests with --skip flag for slow_

Screenshot from 2026-04-10 23-54-33

C] Skipping tests wiht --skip flag for integration

Screenshot from 2026-04-10 23-55-59

@MatthewMckee4 MatthewMckee4 added the cli Related to the command-line interface label Apr 10, 2026
@MatthewMckee4 MatthewMckee4 force-pushed the add-exclude-tests-with-skip-patterns branch from 5d7ae98 to fea3b8c Compare April 10, 2026 18:50
@MatthewMckee4
Copy link
Copy Markdown
Member

MatthewMckee4 commented Apr 10, 2026

You'll need to run cargo run -p karva_dev generate-all to fix those tests!

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Apr 10, 2026

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1 untouched benchmark


Comparing OmChillure:add-exclude-tests-with-skip-patterns (8c2c1c4) with main (3530d3d)1

Open in CodSpeed

Footnotes

  1. No successful run was found on main (482f450) during the generation of this report, so 3530d3d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@OmChillure
Copy link
Copy Markdown
Author

You'll need to run cargo run -p karva_dev generate-all to fix those tests!

You'll need to run cargo run -p karva_dev generate-all to fix those tests!

Hey did this in the latest commit

@MatthewMckee4 MatthewMckee4 force-pushed the add-exclude-tests-with-skip-patterns branch from 26d61a8 to 8c2c1c4 Compare April 10, 2026 23:48
@MatthewMckee4
Copy link
Copy Markdown
Member

Thank you so much for this!

I am going to need to think about this more.

I am quite keep to do something like #546. Meaning we will have a single cli option to filter tests, meaning we can do all three in one.

I do apologise for having this issue open without mentioning this in the issue itself. I don't get PRs very often. So I appreciate the time you've taken to do this, but I don't want to merge this just to then remove the functionality soon after.

Thanks again!

@OmChillure
Copy link
Copy Markdown
Author

OmChillure commented Apr 11, 2026

Thank you so much for this!

I am going to need to think about this more.

I am quite keep to do something like #546. Meaning we will have a single cli option to filter tests, meaning we can do all three in one.

I do apologise for having this issue open without mentioning this in the issue itself. I don't get PRs very often. So I appreciate the time you've taken to do this, but I don't want to merge this just to then remove the functionality soon after.

Thanks again!

Ohh no worries at all ! Thanks for checking it though.

As mentioned in #546, I had a look at the codebase how can we potentailly do those changes,

1. Rewrite the parser in [filter.rs] for the new DSL grammar (&, |, -, not, test(...), tag(...), matcher types)
2. Swap the three CLI flags to a single -E in [lib.rs] 
3. Collapse the three filter types in settings into one expression
4. Simplify the orchestration forwarding in [orchestration.rs]
5. Update all the integration tests 

all these will take a good amount of time.

And seeing your comment on #546,, ig you are thinking on the correct approach? Let me know if I can help here if you have decided to move ahead.

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli Related to the command-line interface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exclude tests with --skip patterns

2 participants