Skip to content

Add mutation testing infrastructure and analysis#34

Merged
sweis merged 1 commit intomainfrom
claude/cargo-mutants-analysis-HZ4wF
Mar 12, 2026
Merged

Add mutation testing infrastructure and analysis#34
sweis merged 1 commit intomainfrom
claude/cargo-mutants-analysis-HZ4wF

Conversation

@sweis
Copy link
Copy Markdown
Owner

@sweis sweis commented Mar 12, 2026

Summary

This PR adds mutation testing infrastructure to the project using cargo-mutants, along with a comprehensive analysis of the baseline mutation test results.

Key Changes

  • Added .cargo/mutants.toml: Configuration for cargo-mutants that:

    • Excludes src/main.rs (CLI with no unit tests) and src/snark.rs (slow Groth16 setup)
    • Skips SNARK tests during mutation runs to reduce runtime from 2+ minutes to ~3 seconds per mutant
    • Excludes semantically equivalent mutations (bitwise operations with non-overlapping patterns)
    • Sets appropriate timeouts for test execution
  • Added notes/cargo-mutants-analysis.md: Detailed analysis of mutation test results (301 total mutants):

    • 67.4% caught (203 mutants) — mutations detected by tests
    • 25.9% missed (78 mutants) — mutations not caught, categorized by root cause
    • 6.3% unviable (19 mutants) — compilation failures
    • 0.3% timeout (1 mutant) — infinite loop, effectively caught
    • Overall mutation score: 72.2%
  • Updated Makefile: Added mutants target to run mutation testing with 2 parallel workers

Notable Findings

The analysis identifies 8 categories of missed mutants:

  1. Groth16-only code paths (26 mutants) — Expected; tests skipped by design
  2. DoS-protection size limits (23 mutants) — Real test gap; no tests with oversized or boundary-value inputs
  3. Claims validation boundaries (9 mutants) — Missing tests for edge cases in Claims::validate()
  4. Dead/untested code (8 mutants) — Including unused Algorithm::signature_len() and generate_hmac_key() not tested
  5. Errors shadowed by later checks (5 mutants) — Low priority; behavior correct but error source not pinned
  6. Key comparison defense gap (1 mutant) — verify_key_match always succeeds in tests; signature check masks the issue
  7. Subtle decoding edge cases (2 mutants) — Proto3 varint overflow and non-minimal encoding edge cases
  8. Semantically equivalent mutations (3 mutants) — Harmless; already excluded in config

Recommendations

Prioritized list for improving mutation score:

  1. Delete unused Algorithm::signature_len()
  2. Add test for validate_public_key_size() with wrong-length keys
  3. Test generate_hmac_key() returns exactly 32 bytes
  4. Assert KeyHashMismatch specifically in wrong-key verify tests
  5. Add Claims::validate boundary tests (255-byte fields)
  6. Add known-value test for bytes_to_fr()
  7. Optional: add oversize-input tests for DoS limits (low priority due to fuzzing coverage)

https://preview.claude.ai/code/session_01DX5NaqQ6ozqtnssna9xuqf

Ran mutation testing on the core library (excluding main.rs and snark.rs
for speed). 301 mutants tested in ~10 minutes; 72.2% mutation score.

Key findings in notes/cargo-mutants-analysis.md:

- Algorithm::signature_len() is dead code (never called anywhere)
- validate_public_key_size() can be replaced with Ok(()) — no test
  passes an Ed25519/ML-DSA-44 VerifyingKey with wrong-length public key
- generate_hmac_key() is only used in main.rs, never tested
- verify_key_match() can be a no-op — tests only check .is_err(),
  sig verification catches the real mismatch (defense-in-depth unproven)
- Claims::validate boundary checks (subject/audience/scope length limits)
  lack tests on the signing side
- DoS-protection upper-bound checks (MAX_PAYLOAD_BYTES etc.) have no
  unit tests that exceed the limit (fuzzing likely covers these)
- Several "missed" mutants are Groth16-only paths, covered when the
  full test suite runs (~26 of the 78 missed)
- 3 semantically-equivalent mutations (|/^ with disjoint bits) added
  to exclude_re

Config in .cargo/mutants.toml, make target: `make mutants`.
@sweis sweis merged commit 4a41b16 into main Mar 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants