Add mutation testing infrastructure and analysis#34
Merged
Conversation
Ran mutation testing on the core library (excluding main.rs and snark.rs for speed). 301 mutants tested in ~10 minutes; 72.2% mutation score. Key findings in notes/cargo-mutants-analysis.md: - Algorithm::signature_len() is dead code (never called anywhere) - validate_public_key_size() can be replaced with Ok(()) — no test passes an Ed25519/ML-DSA-44 VerifyingKey with wrong-length public key - generate_hmac_key() is only used in main.rs, never tested - verify_key_match() can be a no-op — tests only check .is_err(), sig verification catches the real mismatch (defense-in-depth unproven) - Claims::validate boundary checks (subject/audience/scope length limits) lack tests on the signing side - DoS-protection upper-bound checks (MAX_PAYLOAD_BYTES etc.) have no unit tests that exceed the limit (fuzzing likely covers these) - Several "missed" mutants are Groth16-only paths, covered when the full test suite runs (~26 of the 78 missed) - 3 semantically-equivalent mutations (|/^ with disjoint bits) added to exclude_re Config in .cargo/mutants.toml, make target: `make mutants`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds mutation testing infrastructure to the project using
cargo-mutants, along with a comprehensive analysis of the baseline mutation test results.Key Changes
Added
.cargo/mutants.toml: Configuration forcargo-mutantsthat:src/main.rs(CLI with no unit tests) andsrc/snark.rs(slow Groth16 setup)Added
notes/cargo-mutants-analysis.md: Detailed analysis of mutation test results (301 total mutants):Updated
Makefile: Addedmutantstarget to run mutation testing with 2 parallel workersNotable Findings
The analysis identifies 8 categories of missed mutants:
Claims::validate()Algorithm::signature_len()andgenerate_hmac_key()not testedverify_key_matchalways succeeds in tests; signature check masks the issueRecommendations
Prioritized list for improving mutation score:
Algorithm::signature_len()validate_public_key_size()with wrong-length keysgenerate_hmac_key()returns exactly 32 bytesKeyHashMismatchspecifically in wrong-key verify testsClaims::validateboundary tests (255-byte fields)bytes_to_fr()https://preview.claude.ai/code/session_01DX5NaqQ6ozqtnssna9xuqf