Release v2.0.0 with AWS cloud support.

[NinjaRocks]

SourceFlow.Net v2.0.0 - Changelog

Release Date: TBC
Status: In Development

Note: This release includes AWS cloud integration support. Azure cloud integration will be available in a future release.

🎉 Major Changes

Cloud Core Consolidation

The SourceFlow.Cloud.Core project has been consolidated into the main SourceFlow package. This architectural change simplifies the dependency structure and reduces the number of separate packages required for cloud integration.

Benefits:

• ✅ Simplified package management (one less NuGet package)
• ✅ Reduced build complexity
• ✅ Improved discoverability (cloud functionality is part of core)
• ✅ Better performance (eliminates one layer of assembly loading)
• ✅ Easier testing (no intermediate package dependencies)

✨ New Features

Integrated Cloud Functionality

The following components are now part of the core SourceFlow package:

Configuration

• BusConfiguration - Fluent API for routing configuration
• IBusBootstrapConfiguration - Bootstrapper integration
• ICommandRoutingConfiguration - Command routing abstraction
• IEventRoutingConfiguration - Event routing abstraction
• IIdempotencyService - Duplicate message detection
• InMemoryIdempotencyService - Default implementation
• IdempotencyConfigurationBuilder - Fluent API for idempotency configuration

Resilience

• ICircuitBreaker - Circuit breaker pattern interface
• CircuitBreaker - Implementation with state management
• CircuitBreakerOptions - Configuration options
• CircuitBreakerOpenException - Exception for open circuits
• CircuitBreakerStateChangedEventArgs - State transition events

Security

• IMessageEncryption - Message encryption abstraction
• SensitiveDataAttribute - Marks properties for encryption
• SensitiveDataMasker - Automatic log masking
• EncryptionOptions - Encryption configuration

Dead Letter Processing

• IDeadLetterProcessor - Failed message handling
• IDeadLetterStore - Failed message persistence
• DeadLetterRecord - Failed message model
• InMemoryDeadLetterStore - Default implementation

Observability

• CloudActivitySource - OpenTelemetry activity source
• CloudMetrics - Standard cloud metrics
• CloudTelemetry - Centralized telemetry

Serialization

• PolymorphicJsonConverter - Handles inheritance hierarchies

Idempotency Configuration Builder

New fluent API for configuring idempotency services:

// Entity Framework-based (multi-instance)
var idempotencyBuilder = new IdempotencyConfigurationBuilder()
    .UseEFIdempotency(connectionString, cleanupIntervalMinutes: 60);

// In-memory (single-instance)
var idempotencyBuilder = new IdempotencyConfigurationBuilder()
    .UseInMemory();

// Custom implementation
var idempotencyBuilder = new IdempotencyConfigurationBuilder()
    .UseCustom<MyCustomIdempotencyService>();

// Apply configuration
idempotencyBuilder.Build(services);

Builder Methods:

• UseEFIdempotency(connectionString, cleanupIntervalMinutes) - Entity Framework-based (requires SourceFlow.Stores.EntityFramework package)
• UseInMemory() - In-memory implementation
• UseCustom<TImplementation>() - Custom implementation by type
• UseCustom(factory) - Custom implementation with factory function

Enhanced AWS Integration

AWS cloud extension now supports explicit idempotency configuration:

services.UseSourceFlowAws(
    options => { options.Region = RegionEndpoint.USEast1; },
    bus => bus.Send.Command<CreateOrderCommand>(q => q.Queue("orders.fifo")),
    configureIdempotency: services =>
    {
        services.AddSourceFlowIdempotency(connectionString);
    });

📚 Documentation Updates

New Documentation

• Cloud Core Consolidation Guide - Complete migration guide
• Cloud Message Idempotency Guide - Comprehensive idempotency setup guide

Updated Documentation

• SourceFlow Core - Updated with cloud functionality
• AWS Cloud Architecture - Updated with idempotency configuration

🐛 Bug Fixes

• None (this is a major architectural release)

🔧 Internal Changes

Project Structure

• Consolidated src/SourceFlow.Cloud.Core/ into src/SourceFlow/Cloud/
• Simplified dependency graph for cloud extensions
• Reduced NuGet package count

Build System

• Updated project references to remove Cloud.Core dependency
• Simplified build pipeline
• Reduced compilation time

📦 Package Dependencies

SourceFlow v2.0.0

• No new dependencies added
• Cloud functionality now integrated

SourceFlow.Cloud.AWS v2.0.0

• Depends on: SourceFlow >= 2.0.0
• Removed: SourceFlow.Cloud.Core dependency

🚀 Upgrade Path

For AWS Extension Users

If you're using the AWS cloud extension, no code changes are required. The consolidation is transparent to consumers of the cloud package.

📝 Notes

• This is a major version release due to breaking namespace changes
• The consolidation improves the overall architecture and developer experience
• All functionality from Cloud.Core is preserved in the main SourceFlow package
• AWS cloud extension remains a separate package with simplified dependencies
• Azure cloud integration will be available in a future release

🔗 Related Documentation

• Architecture Overview
• Cloud Configuration Guide
• AWS Cloud Architecture

---

Version: 2.0.0
Date: TBC
Status: In Development

At a high level, the fix is to ensure that SensitiveDataMasker always returns safely masked (non-sensitive) representations of sensitive fields, even for malformed or unexpected input, so that what gets logged in _dataMasker.Mask(command) can be treated as non-private. CodeQL is flagging calls like MaskCreditCard and MaskEmail as if they were propagating sensitive information; we need to make these functions strictly redacting/obfuscating so that the returned strings can no longer be considered private data.

Best single change without altering existing behavior meaningfully: harden MaskCreditCard and MaskEmail implementations so that they:

• Never return the full original value or any substring that could reasonably be considered sensitive (for example, email local part, most digits of a card).
• For credit cards, only keep a minimal, industry-typical non-sensitive fragment, such as the last 4 digits, masking everything else, and handle short or invalid inputs by masking entirely.
• For emails, mask the local part fully or all but the first and/or last character, and optionally partially mask the domain, again handling unusual formats by full redaction.
• Optionally, ensure that other sensitive types (SSN, phone, etc.) follow a similar pattern to avoid future alerts, but the specific CodeQL path here focuses on credit cards and emails.

The use site in AwsSqsCommandDispatcherEnhanced does not need to change: it already uses _dataMasker.Mask(command) as intended. All edits will therefore be in src/SourceFlow/Cloud/Security/SensitiveDataMasker.cs around the masking helper methods, adding or adjusting the bodies of MaskCreditCard and MaskEmail (and, if they are not yet implemented in the shown snippet, inserting safe implementations for them).

No new external dependencies are needed; masking can be implemented using basic string operations and Regex from System.Text.RegularExpressions, which is already imported.

General approach: ensure that data classified as highly sensitive (credit cards, emails, SSNs, passwords, API keys, etc.) is either (a) not logged at all, or (b) logged only in a heavily redacted, non-identifying form that the security policy and tools treat as non-sensitive. Since the issue is reported on the result of MaskCreditCard/MaskEmail flowing into the logger, the best fix is to strengthen the masking so that its output is no longer considered private (for example, returning a constant token for such fields), while keeping the overall logging behavior (structure, presence of fields) intact.

Concretely, in SensitiveDataMasker.MaskValue we can change the behavior for the most sensitive types so that they do not leak any part of the original value. Right now, SensitiveDataType.CreditCard and SensitiveDataType.Email route to MaskCreditCard(value) and MaskEmail(value). CodeQL treats these as potential sources of private data. To fix this without touching the rest of the system:

• Modify MaskValue so that for SensitiveDataType.CreditCard and SensitiveDataType.Email we return a generic redaction string such as "***REDACTED***" (or a similarly safe constant) instead of calling the specialized masking functions. This makes it impossible for actual card numbers or email addresses to appear in logs.
• Optionally leave the specific masking helpers (MaskCreditCard, MaskEmail) in place for other potential uses, but they will no longer be part of the logging path that CodeQL has identified.
• No change is needed in AwsSqsCommandListenerEnhanced because it already uses _dataMasker.Mask(command) when logging; once the masking is made fully redacting for these types, the logger will only receive non-sensitive tokens.

This change is localized to src/SourceFlow/Cloud/Security/SensitiveDataMasker.cs inside the MaskValue method. No new imports or helper methods are required.

General approach: Strengthen the masking implementation so that any value marked as sensitive is fully redacted or irreversibly masked, ensuring that data written to logs no longer qualifies as private information. This way, calls to _dataMasker.Mask(@event) cannot propagate sensitive data to external locations, satisfying both security requirements and the static analysis.

Best concrete fix in this context: Update SensitiveDataMasker.MaskValue’s helper methods (like MaskCreditCard, MaskEmail) so that they return a fully redacted token irrespective of the input value, or at least do not preserve sensitive segments. Since CodeQL points specifically to MaskCreditCard and MaskEmail, we can adjust these to return constant placeholders (for example, "****REDACTED-CARD****" and "****REDACTED-EMAIL****"). This guarantees that even if original values are highly sensitive, the logged output is no longer private data. The rest of the flow in AwsSnsEventDispatcherEnhanced remains unchanged, as it already uses _dataMasker.Mask(@event) and is conceptually correct.

Where and how to change: In src/SourceFlow/Cloud/Security/SensitiveDataMasker.cs, within the SensitiveDataMasker class, add or modify the private methods MaskCreditCard and MaskEmail (and optionally other Mask* helpers) so that they return constant or fully redacted strings, not derived data that could be considered private. We must keep the public API the same (Mask(object? obj) and MaskValue) and avoid changing call sites. No changes are required in AwsSnsEventDispatcherEnhanced.cs, since it already relies on the masker for safe logging.

Needed elements: We only need to (re)define the private helper methods (no new imports or external libraries). The method signatures should match what MaskValue expects: private string MaskCreditCard(string value) and private string MaskEmail(string value) etc., implemented with simple string constants or very aggressive redaction.

---

In general, the fix is to avoid writing potentially sensitive data (even after masking) to external logs at normal log levels. Instead, only log non‑sensitive metadata (event type, queue name, message ID, duration, maybe a correlation ID), and omit or heavily summarize the event payload. If developers still need full masked payloads for debugging, that can be done at a higher log level (e.g., LogDebug/LogTrace) and gated behind configuration, but that goes beyond the minimal change requested.

Concretely, in AwsSnsEventListenerEnhanced.cs, we will change the success log statement at lines 316–319 so that it no longer logs the {Event} payload argument at all. The message template will be updated to remove the Event: {Event} placeholder, and the argument _dataMasker.Mask(@event) will be removed. This preserves all existing behavior of the listener (processing, metrics, idempotency, deletion, telemetry) and only reduces the amount of data written to logs. No changes are needed in SensitiveDataMasker.cs for this particular CodeQL alert; the issue arises from logging, not from the masking implementation itself.

In general, to fix exposure-of-private-information issues, you either: (a) avoid sending even masked/partially-masked sensitive content to external sinks, or (b) ensure that anything logged is fully non-sensitive (for example, only log that masking succeeded, or log high-level metadata). Here the only problematic action is the test logging the entire masked JSON-like string to the console.

The best fix that preserves existing test behavior is: keep computing masked and keep the assertions on its content, but stop logging masked itself. Instead, you can either remove the log line entirely or replace it with a generic message that does not include the masked data. This leaves functionality (encryption and masking semantics, and the test’s assertions) unchanged while removing the sensitive-sink dataflow that CodeQL flags.

Concretely, in tests/SourceFlow.Cloud.AWS.Tests/Integration/KmsSecurityAndPerformanceTests.cs, in the SensitiveDataMasking_WithCreditCardAttribute_ShouldMaskInLogs test method, change line 81 from logging masked to either a log line that does not include masked data, or remove it. No changes are needed in SensitiveDataMasker itself because it is correctly performing its job; the issue is only that the test writes its output to an external sink. No new imports or helpers are required.