Skip to content

12 01 Creating A Server

Jump to bottom
Cyberdyne Development edited this page Feb 24, 2026 · 5 revisions

12-01 Creating a Server

This guide covers building an FDW-powered server application using the FractalDataWorks.Hosting and FractalDataWorks.Hosting.MsSql packages. These packages encapsulate the repetitive bootstrap ceremony -- Serilog configuration, ConfigurationDb connection, three-phase ServiceType registration, middleware, and health endpoints -- into a handful of extension methods.

Overview

FDW servers are thin orchestration wrappers around the framework. The hosting extensions encapsulate all the repetitive bootstrap ceremony, and per-domain ServiceTypeCollections provide the runtime behavior. A typical server's Program.cs consists of five framework extension calls plus application-specific wiring (authentication, endpoints, background tasks).

Every FDW server follows the same lifecycle:

WebApplicationBuilder
  1. AddFrameworkSerilog()               ← Two-stage Serilog bootstrap
  2. AddFrameworkConfigurationDb()       ← Connect to ConfigurationDb, load all config
  3. AddFrameworkServiceTypes()          ← Phase 1: Configure + Register ServiceTypeCollections
  4. [custom services]             ← Application-specific DI registrations
  5. builder.Build()

WebApplication
  6. InitializeFrameworkServiceTypes()   ← Phase 2: Initialize (eager resolve, fail-fast)
  7. UseFrameworkMiddleware()            ← Exception handler + HTTPS + security headers + Serilog
  8. [custom middleware]           ← Authentication, CORS, rate limiting
  9. MapFrameworkHealthEndpoint()        ← GET /health
 10. [custom endpoints]            ← FastEndpoints, minimal APIs, SignalR hubs
 11. app.RunAsync()

The hosting packages replace 100-200 lines of manual bootstrap with five extension method calls while preserving full customization between each step.

Minimal Example

A complete FDW server in approximately 20 lines of Program.cs:

using FractalDataWorks.Hosting.Extensions;
using FractalDataWorks.Hosting.MsSql.Extensions;
using Serilog;

var builder = WebApplication.CreateBuilder(args);

var loggerFactory = builder.AddFrameworkSerilog("MyService");
using var configDb = await builder.AddFrameworkConfigurationDb(loggerFactory);

builder.AddFrameworkServiceTypes(loggerFactory, types =>
{
    types.AddSecretManagers()
         .AddConnections()
         .AddDataStores(ds => ds.RegisterMsSql())
         .AddDataSets()
         .AddDataGateway();
});

var app = builder.Build();

app.InitializeFrameworkServiceTypes(loggerFactory);
app.UseFrameworkMiddleware();
app.MapFrameworkHealthEndpoint("MyService");

await app.RunAsync();

This replaces 100-200 lines of manual bootstrap with five extension method calls.

Extension Method Reference

FractalDataWorks.Hosting

Method Extends Returns Description
AddFrameworkSerilog(applicationName, configureBootstrap?) WebApplicationBuilder ILoggerFactory Configures two-stage Serilog bootstrap. Returns a startup ILoggerFactory for MessageLogging during bootstrap.
AddFrameworkServiceTypes(loggerFactory, configure) WebApplicationBuilder WebApplicationBuilder Phase 1 (Configure + Register) for all ServiceTypeCollections via fluent builder.
InitializeFrameworkServiceTypes(loggerFactory) WebApplication WebApplication Phase 2 (Initialize) for all ServiceTypeCollections registered in Phase 1.
UseFrameworkMiddleware(securityHeaders?) WebApplication WebApplication Adds GlobalExceptionHandlerMiddleware, HTTPS redirection, SecurityHeadersMiddleware, and Serilog request logging.
UseInternalApiKeyAuth() IApplicationBuilder IApplicationBuilder Adds InternalApiKeyMiddleware for service-to-service auth via X-Internal-Api-Key header.
UseFrameworkSecurityHeaders(options?) IApplicationBuilder IApplicationBuilder Adds SecurityHeadersMiddleware standalone (use when not calling UseFrameworkMiddleware).
MapFrameworkHealthEndpoint(serviceName) WebApplication void Maps GET /health returning { status, service, timestamp }. Excluded from API documentation.
AddFrameworkCors(configuration) IServiceCollection IServiceCollection Adds CORS policy from the "Cors" section of appsettings.json.

FractalDataWorks.Hosting.MsSql

Method Extends Returns Description
AddFrameworkConfigurationDb(loggerFactory) WebApplicationBuilder Task<ConfigurationDbResult> Bootstraps ConfigurationDb: reads settings from appsettings.json, creates a bootstrap secret manager, connects to SQL Server, and adds the MsSql configuration source.
RegisterMsSql() DataStoreRegistrationBuilder DataStoreRegistrationBuilder Registers MsSqlDataStoreType with the DataStoreTypes MutableTypeCollection.
AddFrameworkMsSqlConfigurationWriterBackend(loggerFactory?) IServiceCollection IServiceCollection Registers the MsSql backend for configuration writers (CQRS write-side).

Parameter Details

AddFrameworkSerilog(applicationName, configureBootstrap?)

Parameter Type Description
applicationName string Used in bootstrap log messages and as the logger source context.
configureBootstrap Action<LoggerConfiguration>? Optional callback to customize the bootstrap logger (e.g., add extra sinks before appsettings.json loads).

The method performs two-stage Serilog initialization:

  • Stage 1 (Bootstrap): Creates a console-only logger with FromLogContext, WithSpan, WithMachineName, and WithEnvironmentName enrichers. This logger is active during startup before appsettings.json is available.
  • Stage 2 (Full): Calls builder.Host.UseSerilog() to read the complete Serilog configuration from appsettings.json after the host is built.

AddFrameworkConfigurationDb(loggerFactory)

Returns ConfigurationDbResult (implements IDisposable):

  • BootstrapConnection (MsSqlConnection) -- the bootstrap connection to ConfigurationDb. Use for downstream registration such as AddSqlAuthentication(configDb.BootstrapConnection.ConnectionString).
  • Always wrap in using to dispose the bootstrap connection.

The method performs five steps internally:

  1. Reads ConfigurationDb section from appsettings.json into a new MsSqlConnectionConfiguration.
  2. Reads the AuthenticationType discriminator (e.g., "SqlAuth") and the Authentication key-value dictionary from the ConfigurationDb section.
  3. Creates a bootstrap EnvironmentVariableSecretManager from the BootstrapSecretManager section for password resolution.
  4. Creates an MsSqlConnection to ConfigurationDb using MsSqlConnectionFactory.
  5. Adds the MsSqlConfigurationSource which loads ALL [ManagedConfiguration] tables in a single Load() call.

Throws InvalidOperationException if the ConfigurationDb or BootstrapSecretManager sections are missing, or if the database connection fails.

Required appsettings.json sections:

{
  "BootstrapSecretManager": {
    "Type": "EnvironmentVariable",
    "Prefix": "FDW_SECRET_",
    "StripPrefix": true
  },
  "ConfigurationDb": {
    "Server": "127.0.0.1",
    "Port": 1433,
    "Database": "ConfigurationDb",
    "DefaultSchema": "cfg",
    "TrustServerCertificate": true,
    "Encrypt": true,
    "AuthenticationType": "SqlAuth",
    "Authentication": {
      "Username": "fdw_config",
      "SecretKeyName": "CONFIG_PASSWORD"
    }
  }
}

The SecretKeyName is resolved via the environment variable FDW_SECRET_CONFIG_PASSWORD.

Discriminator + Dictionary Pattern: The AuthenticationType property selects the authentication strategy (e.g., SqlAuth, WindowsAuth, EntraId, ManagedIdentity). The Authentication section is a flat key-value dictionary — keys and which values are required depend on the selected type. The MsSqlAuthenticationProcessors TypeCollection validates that all required properties are present and builds the connection string accordingly.

UseFrameworkMiddleware(securityHeaders?)

Adds four middleware components in this order:

  1. GlobalExceptionHandlerMiddleware -- Catches unhandled exceptions, returns structured ErrorResponse JSON with correlation ID. Configurable support contact via SupportOptions.
  2. UseHttpsRedirection() -- Redirects HTTP to HTTPS.
  3. SecurityHeadersMiddleware -- Adds OWASP security headers (X-Content-Type-Options, X-Frame-Options, Strict-Transport-Security, Content-Security-Policy, and no-cache headers for sensitive paths).
  4. UseSerilogRequestLogging() -- Structured HTTP request/response logs.

Pass SecurityHeadersOptions to customize behavior:

app.UseFrameworkMiddleware(new SecurityHeadersOptions
{
    AllowFraming = false,          // DENY vs SAMEORIGIN for X-Frame-Options
    EnableDefaultCsp = true,       // Generate default Content-Security-Policy
    ContentSecurityPolicy = null,  // Override CSP string (null = use default)
    SensitivePaths =               // Paths that receive no-cache headers
    [
        "/api/v1/auth",
        "/api/v1/users",
        "/api/v1/tenants"
    ]
});

AddFrameworkCors(configuration)

Reads the "Cors" section from IConfiguration and builds a default CORS policy. When no explicit origins are configured, it allows localhost origins only. Configuration options:

Property Type Default Description
Enabled bool true Master enable/disable for CORS.
Origins IList<string> [] Allowed origins. Empty = localhost only.
Methods IList<string> [GET, POST, PUT, DELETE, PATCH, OPTIONS] Allowed HTTP methods.
Headers IList<string> [Content-Type, Authorization, X-Requested-With, X-Tenant-Id, X-Correlation-Id] Allowed request headers.
ExposedHeaders IList<string> [X-Correlation-Id, X-Request-Id, WWW-Authenticate, X-RateLimit-*] Response headers exposed to the client.
AllowCredentials bool true Whether credentials (cookies, auth headers) are allowed.
PreflightMaxAgeSeconds int 600 How long preflight results are cached (seconds).

MapFrameworkHealthEndpoint(serviceName)

Maps a GET /health endpoint that returns:

{
  "status": "healthy",
  "service": "MyService",
  "timestamp": "2026-02-10T12:00:00Z"
}

The endpoint is excluded from API documentation (Swagger/Scalar).

ServiceTypeRegistrationBuilder

The fluent builder enforces correct dependency ordering for three-phase registration. Each Add*() method performs Phase 1a (Configure -- bind IOptions) and Phase 1b (Register -- register factories and singletons). The builder tracks which domains were registered via ServiceTypeRegistrationState, so that InitializeFrameworkServiceTypes() only initializes what was actually registered.

Required Order

The first four methods must be called in dependency order:

AddSecretManagers()   ← Must be first (provides password resolution)
       |
AddConnections()      ← Depends on SecretManagers for secret resolution
       |
AddDataStores(...)    ← Depends on Connections for database access
       |
AddDataSets()         ← Depends on DataStores for physical storage

Then Any Order

After the first four, the remaining methods can be called in any order:

AddAuthentication()        ← Depends on Connections
AddAuthorization()         ← Depends on Authentication
AddEtlPipelines()          ← Depends on DataStores + DataSets
AddSchedulers()            ← Depends on DataStores
AddConfigurationWriters()  ← Depends on Connections + DataStores
AddDataGateway()           ← Depends on DataStores + DataSets

Full Builder Reference

Method Service Domain What It Registers
AddSecretManagers() SecretManagers Password resolution providers (EnvironmentVariable, AzureKeyVault, UserSecrets).
AddConnections() Connections Database connection factories with secret resolution. Also registers IDataConnectionProvider via RegisterAdditionalInterfaces.
AddDataStores(configure?) DataStores Data store providers. Pass a callback to register cross-assembly types (e.g., ds => ds.RegisterMsSql()).
AddDataSets() DataSets Logical data definitions bound to physical DataStores.
AddAuthentication() Authentication JWT, Basic, OAuth2, and other authentication types.
AddAuthorization() Authorization RBAC authorization bridge. Registers FdwAuthorizationPolicyProvider and FrameworkPermissionHandler for fdw:{resource}:{action} endpoint policies. See 12-05 Authorization.
AddEtlPipelines() ETL Pipeline execution with transforms and row sources.
AddSchedulers() Scheduling Cron and interval job scheduling.
AddConfigurationWriters() Configuration CQRS write-side for configuration (used with AddFrameworkMsSqlConfigurationWriterBackend).
AddDataGateway() DataGateway Unified data access layer (IDataGateway). No Configure/Register/Initialize phases -- just registers services.

DataStore Registration Callback

DataStoreTypes is a MutableTypeCollection. The MsSql DataStore type lives in a separate assembly (FractalDataWorks.Data.DataStores.SqlServer) and must be registered explicitly through the callback:

.AddDataStores(ds => ds.RegisterMsSql())

The RegisterMsSql() extension method (from FractalDataWorks.Hosting.MsSql) calls DataStoreTypes.RegisterMember(new MsSqlDataStoreType()) before the Configure/Register phases run.

You can register additional custom DataStore types through the same callback:

.AddDataStores(ds =>
{
    ds.RegisterMsSql();
    ds.Register(() => DataStoreTypes.RegisterMember(new MyCustomDataStoreType()));
})

What Each Method Does Internally

Each Add*() method performs two operations on the corresponding ServiceTypeCollection:

AddConnections()
  1. ConnectionTypes.Configure(builder, loggerFactory)
       → Each ServiceTypeOption.Configure() binds IOptions<List<T>> from IConfiguration
  2. ConnectionTypes.Register(services, loggerFactory)
       → Each ServiceTypeOption.RegisterRequiredServices() registers factories
  3. ConnectionTypes.RegisterAdditionalInterfaces(services)
       → Registers IDataConnectionProvider for cross-domain access

The state is stored in ServiceTypeRegistrationState (added to DI as a singleton). When InitializeFrameworkServiceTypes() runs after Build(), it reads this state and calls {Domain}Types.Initialize(services, loggerFactory) only for domains that were registered:

InitializeFrameworkServiceTypes()
  For each registered domain:
    1. {Domain}Types.Initialize(services, loggerFactory)
         → Each ServiceTypeOption.RegisterFactory() wires factory into provider
    2. DataStoreTypes.RegisterContainers(services, loggerFactory)  [DataStores only]
         → Registers DataContainer instances for schema-aware access

Customization Points

Adding Custom Services

Register application-specific services between AddFrameworkServiceTypes and builder.Build():

builder.AddFrameworkServiceTypes(loggerFactory, types => { ... });

// Kestrel request limits
builder.WebHost.ConfigureKestrel(options =>
{
    options.Limits.MaxRequestBodySize = 10 * 1024 * 1024; // 10 MB
    options.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(30);
});

// Graceful shutdown
builder.Services.Configure<HostOptions>(o => o.ShutdownTimeout = TimeSpan.FromSeconds(30));

// Application services
builder.Services.AddHttpClient();
builder.Services.AddScoped<IMyService, MyService>();
builder.Services.Configure<MyOptions>(builder.Configuration.GetSection("MyOptions"));

// Execution tracking (if using IDataGateway for ops tables)
builder.Services.AddSingleton<IExecutionTracker>(sp =>
    new ExecutionTrackingService(
        sp.GetRequiredService<IDataGateway>(),
        sp.GetRequiredService<ILoggerFactory>(),
        "OpsDb"));

var app = builder.Build();

Adding Custom Middleware

Add middleware between UseFrameworkMiddleware and endpoint mapping. Middleware order matters:

app.InitializeFrameworkServiceTypes(loggerFactory);

// FDW standard middleware (exception handler + HTTPS + security headers + Serilog logging)
app.UseFrameworkMiddleware();

// CORS -- must be before authentication for preflight (OPTIONS) to work
app.UseCors();

// Authentication and authorization
app.UseAuthentication();
app.UseAuthorization();

// Multi-tenancy (from FractalDataWorks.Services.Multitenancy.Sql)
app.UseMultitenancy();
app.UseRateLimiter();

// Internal API key auth (for backend services)
app.UseInternalApiKeyAuth();

// Endpoints
app.MapFrameworkHealthEndpoint("MyService");

Adding Endpoints

FDW servers commonly use FastEndpoints or minimal APIs for custom endpoints:

// FastEndpoints (register before Build)
builder.Services.AddFastEndpoints();

// After Build
app.UseFastEndpoints(config => config.Endpoints.RoutePrefix = "api/v1");

// Minimal APIs
app.MapGet("/api/status", () => Results.Ok(new { version = "1.0" }));

// SignalR hubs
app.MapHub<PipelineStatusHub>("/hubs/pipelines");

Adding CORS

// Before Build
builder.Services.AddFrameworkCors(builder.Configuration);

// After Build, before authentication
var corsOptions = app.Services.GetService<CorsOptions>();
if (corsOptions?.Enabled == true)
{
    app.UseCors();
}

With appsettings.json:

{
  "Cors": {
    "Enabled": true,
    "Origins": ["https://localhost:5007"],
    "Methods": ["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"],
    "Headers": ["Content-Type", "Authorization", "X-Tenant-Id"],
    "AllowCredentials": true,
    "PreflightMaxAgeSeconds": 600
  }
}

Internal API Key Authentication

For backend services that only accept requests from the API gateway or other internal services:

// Before Build
builder.Services.Configure<InternalApiKeyOptions>(
    builder.Configuration.GetSection(InternalApiKeyOptions.SectionName));

// After Build
app.UseInternalApiKeyAuth();

With appsettings.json:

{
  "InternalApi": {
    "ApiKey": "dev-internal-api-key-change-in-production",
    "HeaderName": "X-Internal-Api-Key"
  }
}

Callers must include X-Internal-Api-Key: <key> in all requests. The HeaderName defaults to X-Internal-Api-Key but can be customized.

Using the ConfigurationDb Bootstrap Connection

The ConfigurationDbResult.BootstrapConnection is available for downstream registration that needs a direct SQL connection before the DI container is built:

using var configDb = await builder.AddFrameworkConfigurationDb(loggerFactory);

// Use bootstrap connection for SQL-based auth registration
builder.Services.AddSqlAuthentication(configDb.BootstrapConnection.ConnectionString);

Fail-Fast Configuration Validation

Database-loaded configuration must be validated at startup. Never fall back to empty defaults:

var app = builder.Build();
app.InitializeFrameworkServiceTypes(loggerFactory);

// Required config -- fail with structured MessageLogging code
var jwtConfig = builder.Configuration.GetSection("Authentication:Jwt")
    .Get<List<JwtAuthenticationConfiguration>>()?.FirstOrDefault();

if (jwtConfig is null)
{
    BootstrapLog.JwtConfigurationMissing(bootstrapLogger);
    return 1;   // Fail fast with exit code
}

// Optional feature -- make registration conditional
var tenantConfig = builder.Configuration.GetSection("TenantProviders")
    .Get<List<SqlTenantConfiguration>>()?.FirstOrDefault();

if (tenantConfig is not null)
{
    // Register tenant provider and middleware
}

Migration Guide

Before: Manual Bootstrap (~100 lines)

// Manual Serilog setup
Log.Logger = new LoggerConfiguration()
    .MinimumLevel.Information()
    .WriteTo.Console()
    .CreateBootstrapLogger();

using var bootstrapLoggerFactory = new SerilogLoggerFactory(Log.Logger);

// Manual ConfigurationDb bootstrap
var configDbConfig = builder.Configuration
    .GetSection("ConfigurationDb")
    .Get<MsSqlConnectionConfiguration>()!;

var secretManagerFactory = new EnvironmentVariableSecretManagerFactory(
    bootstrapLoggerFactory.CreateLogger<EnvironmentVariableSecretManagerFactory>());
var secretManagerResult = await secretManagerFactory.CreateSecretManager(
    new EnvironmentVariableConfiguration());
var secretManager = secretManagerResult.Value;

var connectionFactory = new MsSqlConnectionFactory(
    bootstrapLoggerFactory.CreateLogger<MsSqlConnectionFactory>(),
    bootstrapLoggerFactory.CreateLogger<MsSqlConnection>());
var connectionResult = await connectionFactory.Create(
    configDbConfig, secretManager, bootstrapLoggerFactory);

using var bootstrapConnection = (MsSqlConnection)connectionResult.Value!;
builder.Configuration.AddMsSqlConfiguration(bootstrapConnection, loggerFactory);

// Manual three-phase registration
SecretManagerTypes.Configure(builder, bootstrapLoggerFactory);
SecretManagerTypes.Register(builder.Services, bootstrapLoggerFactory);
ConnectionTypes.Configure(builder, bootstrapLoggerFactory);
ConnectionTypes.Register(builder.Services, bootstrapLoggerFactory);
ConnectionTypes.RegisterAdditionalInterfaces(builder.Services);
DataStoreTypes.RegisterMember(new MsSqlDataStoreType());
DataStoreTypes.Configure(builder, bootstrapLoggerFactory);
DataStoreTypes.Register(builder.Services, bootstrapLoggerFactory);

var app = builder.Build();

SecretManagerTypes.Initialize(app.Services, bootstrapLoggerFactory);
ConnectionTypes.Initialize(app.Services, bootstrapLoggerFactory);
DataStoreTypes.Initialize(app.Services, bootstrapLoggerFactory);
DataStoreTypes.RegisterContainers(app.Services, bootstrapLoggerFactory);

After: Hosting Extensions (~15 lines)

var loggerFactory = builder.AddFrameworkSerilog("MyService");
using var configDb = await builder.AddFrameworkConfigurationDb(loggerFactory);

builder.AddFrameworkServiceTypes(loggerFactory, types =>
{
    types.AddSecretManagers()
         .AddConnections()
         .AddDataStores(ds => ds.RegisterMsSql())
         .AddDataSets()
         .AddDataGateway();
});

var app = builder.Build();
app.InitializeFrameworkServiceTypes(loggerFactory);

Migration Steps

  1. Add NuGet packages:

    dotnet add package FractalDataWorks.Hosting
dotnet add package FractalDataWorks.Hosting.MsSql

  2. Replace Serilog bootstrap with builder.AddFrameworkSerilog("ServiceName"). Remove manual LoggerConfiguration, CreateBootstrapLogger, and SerilogLoggerFactory calls.

  3. Replace ConfigurationDb bootstrap with await builder.AddFrameworkConfigurationDb(loggerFactory). Remove manual MsSqlConnectionConfiguration binding, EnvironmentVariableSecretManagerFactory, MsSqlConnectionFactory, and AddMsSqlConfiguration calls.

  4. Replace manual registration calls with builder.AddFrameworkServiceTypes(...) fluent builder. Remove all individual {Domain}Types.Configure() and {Domain}Types.Register() calls.

  5. Replace manual Initialize calls with app.InitializeFrameworkServiceTypes(loggerFactory). Remove all individual {Domain}Types.Initialize() calls.

  6. Replace middleware setup with app.UseFrameworkMiddleware(). Remove manual UseHttpsRedirection, SecurityHeadersMiddleware, GlobalExceptionHandlerMiddleware, and UseSerilogRequestLogging calls if they match the default pipeline.

  7. Replace health endpoint with app.MapFrameworkHealthEndpoint("ServiceName").

  8. Verify appsettings.json has the required BootstrapSecretManager and ConfigurationDb sections.

  9. Wrap in try/catch/finally for structured error handling:

    try
{
    // ... bootstrap and run ...
    return 0;
}
catch (Exception ex)
{
    Log.Fatal(ex, "Service terminated unexpectedly");
    return 1;
}
finally
{
    await Log.CloseAndFlushAsync().ConfigureAwait(false);
}

Real-World Examples

The Reference Solutions demonstrate three server configurations with increasing complexity:

EtlServer (Simplest -- ~97 lines)

Internal backend service for pipeline execution. Uses API key authentication, no CORS, no JWT.

var loggerFactory = builder.AddFrameworkSerilog("Reference.Etl.Server");
using var configDb = await builder.AddFrameworkConfigurationDb(loggerFactory);

builder.AddFrameworkServiceTypes(loggerFactory, types =>
{
    types.AddSecretManagers()
         .AddConnections()
         .AddDataStores(ds => ds.RegisterMsSql())
         .AddDataSets()
         .AddEtlPipelines()
         .AddDataGateway();
});

// Internal API key auth + execution tracking
builder.Services.Configure<InternalApiKeyOptions>(
    builder.Configuration.GetSection(InternalApiKeyOptions.SectionName));
builder.Services.AddSingleton<IExecutionTracker>(sp =>
    new ExecutionTrackingService(
        sp.GetRequiredService<IDataGateway>(),
        sp.GetRequiredService<ILoggerFactory>(),
        "OpsDb"));

var app = builder.Build();
app.InitializeFrameworkServiceTypes(loggerFactory);
app.UseFrameworkMiddleware();
app.UseInternalApiKeyAuth();
app.MapFrameworkHealthEndpoint("Reference.Etl.Server");
app.MapTriggerEndpoints();

Source: ReferenceSolutions/EtlServer/src/Reference.Etl.Server/Program.cs

SchedulerServer (Moderate -- ~117 lines)

Internal backend service for job scheduling. Adds configuration writers and fail-fast validation.

builder.AddFrameworkServiceTypes(loggerFactory, types =>
{
    types.AddSecretManagers()
         .AddConnections()
         .AddDataStores(ds => ds.RegisterMsSql())
         .AddDataSets()
         .AddSchedulers()
         .AddConfigurationWriters()
         .AddDataGateway();
});

builder.Services.AddFrameworkMsSqlConfigurationWriterBackend(loggerFactory);

var app = builder.Build();
app.InitializeFrameworkServiceTypes(loggerFactory);

// Fail-fast: validate scheduler configuration loaded from database
var schedulerConfig = app.Services
    .GetService<IOptionsMonitor<List<SchedulerConfiguration>>>()?.CurrentValue?.FirstOrDefault();

if (schedulerConfig is null)
{
    BootstrapLog.SchedulerConfigurationMissing(bootstrapLogger);
    return 1;
}

Source: ReferenceSolutions/SchedulerServer/src/Reference.Scheduler.Server/Program.cs

ApiSolution (Full-Featured -- ~497 lines)

Public-facing API with JWT authentication, CORS, OpenTelemetry, multi-tenancy, rate limiting, SignalR, and Scalar API documentation.

builder.AddFrameworkServiceTypes(loggerFactory, types =>
{
    types.AddSecretManagers()
         .AddConnections()
         .AddDataStores(ds => ds.RegisterMsSql())
         .AddDataSets()
         .AddAuthentication()
         .AddAuthorization()
         .AddEtlPipelines()
         .AddConfigurationWriters()
         .AddDataGateway();
});

// OpenTelemetry, JWT, CORS, FastEndpoints, SignalR, rate limiting, multi-tenancy...
builder.Services.AddFrameworkCors(builder.Configuration);
builder.Services.AddFastEndpoints();
builder.Services.AddFrameworkRateLimiting(loggerFactory);

var app = builder.Build();
app.InitializeFrameworkServiceTypes(loggerFactory);

app.UseFrameworkMiddleware(securityHeadersOptions);
app.UseCors();
app.UseAuthentication();
app.UseAuthorization();
app.UseFastEndpoints(config => config.Endpoints.RoutePrefix = "api/v1");

Source: ReferenceSolutions/ApiSolution/src/Reference.Api/Program.cs

Summary Comparison

Aspect EtlServer SchedulerServer ApiSolution
Auth API key API key JWT Bearer
CORS No No Yes
ServiceTypes 6 7 8
ConfigWriters No Yes Yes
Endpoints Minimal API (triggers) Minimal API (schedules) FastEndpoints + Scalar
SignalR No No Yes (3 hubs)
Multi-tenancy No No Yes
Lines of Code ~97 ~117 ~497

Creating Each Server Type

The three Reference Solution servers demonstrate increasing complexity. All follow the same thin-client pattern: FDW provides the service domains, hosting, and data access; the server provides application-specific orchestration.

Creating an API Server

An API server is the public-facing entry point. It registers all service domains, exposes REST endpoints via FastEndpoints, and handles authentication/authorization.

Required packages:

<PackageReference Include="FractalDataWorks.Hosting" />
<PackageReference Include="FractalDataWorks.Hosting.MsSql" />
<PackageReference Include="FractalDataWorks.Services.Connections.Endpoints" />
<PackageReference Include="FractalDataWorks.Services.Data.Endpoints" />
<!-- Add per-domain .Endpoints packages for each domain you expose -->
<PackageReference Include="FastEndpoints" />

Key steps:

  1. Bootstrap with AddFrameworkSerilog + AddFrameworkConfigurationDb + AddFrameworkServiceTypes
  2. Register all needed service domains (SecretManagers, Connections, DataStores, DataSets, Authentication, Authorization, etc.)
  3. Add FastEndpoints, CORS, rate limiting, and SignalR hubs as needed
  4. Create thin endpoint closures over the per-domain .Endpoints base classes (see 13-02 Creating Consumer Packages)
  5. Add Scalar API documentation

Reference: ReferenceSolutions/ApiSolution/src/Reference.Api/Program.cs (~497 lines)

Creating an ETL Server

An ETL server is an internal backend service for pipeline execution. It uses API key authentication (no JWT), no CORS, and exposes trigger endpoints for pipeline dispatch.

Required packages:

<PackageReference Include="FractalDataWorks.Hosting" />
<PackageReference Include="FractalDataWorks.Hosting.MsSql" />
<!-- ETL pipelines are registered via AddEtlPipelines() -->

Key steps:

  1. Bootstrap with the standard five extension calls
  2. Register: SecretManagers, Connections, DataStores, DataSets, EtlPipelines, DataGateway
  3. Add InternalApiKeyOptions for service-to-service auth
  4. Add IExecutionTracker for pipeline execution tracking via IDataGateway
  5. Map trigger endpoints (minimal APIs that accept pipeline execution requests)
  6. Use UseInternalApiKeyAuth() middleware

What stays thin: The ETL server's custom code (~1,541 LOC) is pipeline dispatch orchestration and trigger endpoint mapping. All transform logic, row source resolution, and data access come from FDW's EtlPipelineTypes ServiceTypeCollection.

Reference: ReferenceSolutions/EtlServer/src/Reference.Etl.Server/Program.cs (~97 lines)

Creating a Scheduler Server

A Scheduler server evaluates schedules on a timer and dispatches work to the ETL server. It uses configuration writers to update schedule state.

Required packages:

<PackageReference Include="FractalDataWorks.Hosting" />
<PackageReference Include="FractalDataWorks.Hosting.MsSql" />
<PackageReference Include="FractalDataWorks.Configuration.Writers" />

Key steps:

  1. Bootstrap with the standard five extension calls
  2. Register: SecretManagers, Connections, DataStores, DataSets, Schedulers, ConfigurationWriters, DataGateway
  3. Add AddFrameworkMsSqlConfigurationWriterBackend for schedule state persistence
  4. Validate scheduler configuration with fail-fast (BootstrapLog.SchedulerConfigurationMissing + return 1)
  5. Implement a BackgroundService that evaluates cron/interval triggers using FDW SchedulerTypes
  6. Use HttpClient to dispatch execution requests to the ETL server via its trigger endpoints

What stays thin: The Scheduler server's custom code (~2,039 LOC) is schedule evaluation logic and dispatch orchestration. All schedule type resolution, cron parsing, and configuration management come from FDW's SchedulerTypes ServiceTypeCollection and IConfigurationWriter.

Reference: ReferenceSolutions/SchedulerServer/src/Reference.Scheduler.Server/Program.cs (~117 lines)

Creating a Management UI

See 11-01 Management UI Overview for detailed guidance on creating MudBlazor, Tailwind, and WASM variants. All three are rendering-only skins consuming FDW's UI.Components.Blazor Protocol providers.

Related Documentation

Troubleshooting

ConfigurationDb Connection Failures

Symptom: Application crashes at startup with InvalidOperationException from AddFrameworkConfigurationDb.

Common causes:

  1. SQL Server not running -- check docker ps for the SQL container
  2. Wrong credentials -- verify FDW_SECRET_CONFIG_PASSWORD environment variable
  3. Database doesn't exist -- deploy dacpac first
  4. Wrong port -- check ConfigurationDb.Port in appsettings.json matches Docker mapping

Missing Secret Manager

Symptom: BootstrapSecretManager section is missing or empty

Fix: Add the BootstrapSecretManager section to appsettings.json:

"BootstrapSecretManager": {
    "Type": "EnvironmentVariable",
    "Prefix": "FDW_SECRET_",
    "StripPrefix": true
}

JWT Configuration Missing

Symptom: API starts but all authenticated endpoints return 401.

Fix: Ensure JWT authentication is configured in ConfigurationDb (cfg.Authentication + cfg.JwtAuthentication tables) and the API's service domain registration includes .AddAuthentication().

Port Conflicts

Symptom: Address already in use on startup.

Fix: Check for other services using the same port. Default ports:

  • API: 5001
  • Scheduler: 5000
  • ETL: 5002
  • UI: 5003

ServiceType Registration Order

Symptom: InvalidOperationException during InitializeFrameworkServiceTypes mentioning a missing service.

Fix: Ensure registration order follows: SecretManagers -> Connections -> DataStores -> DataSets -> [others]

Clone this wiki locally