fix: check key type in C_SignInit/C_VerifyInit to prevent crash

[mingulov]

Passing a wrong asymmetric key type (e.g. EC key) to a mismatched mechanism (for example RSA related) caused a crash (segfault or abort) because the code tried to read RSA attributes from an EC key object and passed invalid data to OpenSSL.

Add key type validation after the mechanism switch in both functions, returning CKR_KEY_TYPE_INCONSISTENT on mismatch. Also convert the GOST catch-all else branch to an explicit else-if(isGOST) check.

Regression tests use cross-matched asymmetric keys (EC key with RSA mechanisms, RSA key with ECDSA/EdDSA mechanisms) to verify the fix. Without the fix, C_SignInit accepts the wrong key type and the subsequent C_Sign crashes.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced key type validation for signing and verification operations to ensure consistency with the cryptographic algorithm being used.

• Tests

  • Added test coverage for key type mismatch scenarios during sign and verify initialization.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR modifies asymmetric sign/verify initialization in SoftHSM to validate that the selected mechanism family (RSA, DSA, ECC, GOST, EdDSA) matches the retrieved key type, and adds test cases verifying this validation returns CKR_KEY_TYPE_INCONSISTENT for mismatches.

Changes

Cohort / File(s)	Summary
Core Validation Logic src/lib/SoftHSM.cpp	Modified AsymSignInit and AsymVerifyInit to fetch key metadata upfront, introduce an isGOST flag, and add consolidated key-type validation after mechanism selection that returns CKR_KEY_TYPE_INCONSISTENT when mechanism family does not match retrieved key type.
Test Coverage src/lib/test/SignVerifyTests.cpp, src/lib/test/SignVerifyTests.h	Added two new test methods (testSignInitWrongKeyType and testVerifyInitWrongKeyType) with CppUnit registrations that verify sign/verify initialization fails with CKR_KEY_TYPE_INCONSISTENT when mechanism and key type are mismatched.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• softhsm/SoftHSMv2#683: Modifies asymmetric sign/verify mechanism handling and introduces mechanism-family flags (ECDSA) with overlapping code paths.

Suggested reviewers

• bjosv
• jschlyter
• kalvdans

Poem

🐰 A rabbit hops through keytypes with glee,
Matching mechanisms to what they should be!
No more mismatches in sign and verify,
With validation in place, the checks multiply ✨
GOST flags and logic make crypto quite neat!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly describes the main change: adding key type validation in C_SignInit/C_VerifyInit to prevent crashes from mismatched key types.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/test/SignVerifyTests.cpp (1)

1290-1439: Optional: extract shared “wrong key type” assertion helper to reduce duplication.

Both new tests repeat the same mechanism-loop/assert logic; a small helper would keep future mechanism list updates in one place.

Refactor sketch

+static void assertInitWrongKeyType(
+	CK_SESSION_HANDLE hSession,
+	CK_OBJECT_HANDLE hKey,
+	const CK_MECHANISM_TYPE* mechs,
+	size_t mechCount,
+	CK_RV (*initFn)(CK_SESSION_HANDLE, CK_MECHANISM_PTR, CK_OBJECT_HANDLE))
+{
+	for (size_t i = 0; i < mechCount; i++)
+	{
+		CK_MECHANISM mechanism = { mechs[i], NULL_PTR, 0 };
+		CK_RV rv = initFn(hSession, &mechanism, hKey);
+		CPPUNIT_ASSERT(rv == CKR_KEY_TYPE_INCONSISTENT);
+	}
+}

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/lib/test/SignVerifyTests.cpp` around lines 1290 - 1439, Both tests
testSignInitWrongKeyType and testVerifyInitWrongKeyType duplicate the same
mechanism-loop/assert pattern; extract a small helper (e.g.,
assertInitReportsWrongKeyType) that takes a session handle, a mechanism type
array + count, a key handle and a pointer/enum to indicate whether to call
C_SignInit or C_VerifyInit, iterates the mechanisms, builds CK_MECHANISM, calls
the appropriate PKCS#11 init function (C_SignInit or C_VerifyInit) and asserts
CKR_KEY_TYPE_INCONSISTENT; move the mech arrays (rsaMechs, dsaMechs, ecdsaMechs)
to local/shared constants and replace the duplicated loops in
testSignInitWrongKeyType and testVerifyInitWrongKeyType with calls to this
helper, keeping calls to generateEC and generateRSA and existing key handles
(hEcPub/hEcPriv, hRsaPub/hRsaPriv) unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/lib/test/SignVerifyTests.cpp`:
- Around line 1290-1439: Both tests testSignInitWrongKeyType and
testVerifyInitWrongKeyType duplicate the same mechanism-loop/assert pattern;
extract a small helper (e.g., assertInitReportsWrongKeyType) that takes a
session handle, a mechanism type array + count, a key handle and a pointer/enum
to indicate whether to call C_SignInit or C_VerifyInit, iterates the mechanisms,
builds CK_MECHANISM, calls the appropriate PKCS#11 init function (C_SignInit or
C_VerifyInit) and asserts CKR_KEY_TYPE_INCONSISTENT; move the mech arrays
(rsaMechs, dsaMechs, ecdsaMechs) to local/shared constants and replace the
duplicated loops in testSignInitWrongKeyType and testVerifyInitWrongKeyType with
calls to this helper, keeping calls to generateEC and generateRSA and existing
key handles (hEcPub/hEcPriv, hRsaPub/hRsaPriv) unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 47c75ed5-e53c-4a82-ad36-51d68e185e45

📥 Commits

Reviewing files that changed from the base of the PR and between 3e6e6c3 and ce64feb.

📒 Files selected for processing (3)

• src/lib/SoftHSM.cpp
• src/lib/test/SignVerifyTests.cpp
• src/lib/test/SignVerifyTests.h