17 01 Build Pipeline Guide

Build Pipeline Guide

This guide explains the FractalDataWorks build pipeline -- from building and packing the core framework, through SharedPackages, to all ReferenceSolutions.

Overview

The build pipeline follows a strict dependency chain:

FDW Framework (public/) --> SharedPackages (Reference.*) --> ReferenceSolutions
       |                        |                               |
   pack-local.sh           pack-shared.sh                  build-all.sh

FDW Framework: Built and packed into NuGet packages in LocalNugetFolder
SharedPackages: Cross-cutting packages (e.g., Reference.Auth) that depend on FDW packages
ReferenceSolutions: Four server applications that consume both FDW and SharedPackages via NuGet

All builds use local NuGet packages rather than published feeds, allowing framework changes to propagate immediately to consumer projects.

1. build-all.sh Walkthrough (Framework-Level)

The primary build script at scripts/build-all.sh orchestrates the entire pipeline in three phases.

Usage:

./scripts/build-all.sh                  # Build everything (Debug)
./scripts/build-all.sh -c Release       # Build everything (Release)

Prerequisites:

LocalNugetFolder environment variable must be set
Nerdbank.GitVersioning (nbgv) installed as a dotnet tool

Phase 1: Build and Pack FDW Framework

The script builds the framework solution, then packs all projects into NuGet packages:

# From build-all.sh (Phase 1)
dotnet build "$ROOT_DIR/FractalDataWorks.DeveloperKit.slnx" -c "$CONFIGURATION"

# Detect version via nbgv
VERSION=$(nbgv get-version -v NuGetPackageVersion 2>/dev/null || true)

# Clean old FDW packages from local folder and NuGet cache
find "$LocalNugetFolder" -maxdepth 1 -name "FractalDataWorks.*.nupkg" -delete 2>/dev/null || true
CACHE_ROOT="$HOME/.nuget/packages"
for pkg_dir in "$CACHE_ROOT"/fractaldataworks.*; do
    [ -d "$pkg_dir" ] && rm -rf "$pkg_dir"
done

# Pack (reuses build output)
dotnet pack "$ROOT_DIR/FractalDataWorks.DeveloperKit.slnx" -c "$CONFIGURATION" -o "$LocalNugetFolder" \
    -p:IsPackable=true --no-build

If nbgv is not available, the version is detected from the packed package filename:

ACTUAL_PKG=$(ls "$LocalNugetFolder"/FractalDataWorks.Abstractions.*.nupkg 2>/dev/null | head -1)
VERSION=$(basename "$ACTUAL_PKG" .nupkg | sed 's/^FractalDataWorks\.Abstractions\.//')

Phase 2: Build and Pack SharedPackages

SharedPackages are cross-cutting libraries named Reference.* (e.g., Reference.Auth, Reference.Nfl.Data). The script updates their FdwVersion to match the freshly packed framework, then builds and packs them:

# Update FdwVersion in SharedPackages Directory.Packages.props
SHARED_PROPS="$REF_DIR/SharedPackages/Directory.Packages.props"
sed -i "s|<FdwVersion>[^<]*</FdwVersion>|<FdwVersion>$VERSION</FdwVersion>|g" "$SHARED_PROPS"

# Build and pack
dotnet build "$REF_DIR/SharedPackages/SharedPackages.slnx" -c "$CONFIGURATION"
dotnet pack "$REF_DIR/SharedPackages/SharedPackages.slnx" -c "$CONFIGURATION" -o "$LocalNugetFolder" \
    -p:Version="$VERSION" -p:PackageVersion="$VERSION" \
    -p:IsPackable=true -p:NerdbankGitVersioningPackageVersionEnabled=false --no-build

Phase 3: Build All ReferenceSolutions

The script updates FdwVersion in every consumer Directory.Packages.props file, then builds each solution with a forced NuGet restore:

# Update FdwVersion in all consumer Directory.Packages.props
find "$REF_DIR" -name "Directory.Packages.props" -not -path "*/bin/*" \
    -not -path "*/obj/*" -not -path "*/SharedPackages/*" | while read -r props_file; do
    if grep -q '<FdwVersion>' "$props_file"; then
        sed -i "s|<FdwVersion>[^<]*</FdwVersion>|<FdwVersion>$VERSION</FdwVersion>|g" "$props_file"
    fi
done

# Build solutions with .slnx files
for slnx in "$REF_DIR"/ApiSolution/ApiSolution.slnx \
             "$REF_DIR"/EtlServer/EtlServer.slnx \
             "$REF_DIR"/SchedulerServer/SchedulerServer.slnx \
             "$REF_DIR"/ManagementUI/ManagementUI.slnx; do
    dotnet restore "$slnx" --force 2>&1
    dotnet build "$slnx" -c "$CONFIGURATION" --no-restore 2>&1
done

Standalone projects (ManagementUI-Tailwind, ManagementUI-WASM) are built individually by .csproj path.

2. Version Propagation

Versions flow from Nerdbank.GitVersioning through the entire pipeline via the FdwVersion MSBuild property.

Version Detection

The source of truth is version.json at the repo root, consumed by Nerdbank.GitVersioning:

# Detect current version
nbgv get-version -v NuGetPackageVersion
# Example output: 0.6.42-beta

Version Flow

version.json (repo root)
    |
    v
nbgv get-version --> VERSION variable
    |
    +---> LocalNugetFolder/FdwVersion.props
    |         <FdwVersion>0.6.42-beta</FdwVersion>
    |
    +---> LocalNugetFolder/versions.json
    |         {"FractalDataWorks": "0.6.42-beta"}
    |
    +---> SharedPackages/Directory.Packages.props
    |         <FdwVersion>0.6.42-beta</FdwVersion>
    |
    +---> ApiSolution/Directory.Packages.props
    +---> EtlServer/Directory.Packages.props
    +---> SchedulerServer/Directory.Packages.props
    +---> ManagementUI/Directory.Packages.props
              <FdwVersion>0.6.42-beta</FdwVersion>

The pack-local.sh script generates FdwVersion.props and versions.json in LocalNugetFolder, which downstream scripts read to stay in sync.

3. NuGet Cache Clearing

Both build-all.sh and pack-local.sh aggressively clear cached packages to prevent stale versions. This is critical because local package versions may not change between builds (same git height produces the same version).

What gets cleared:

Location	Pattern	Why
`$LocalNugetFolder`	`FractalDataWorks.*.nupkg`	Old framework packages
`$LocalNugetFolder`	`Reference.*.nupkg`	Old SharedPackages
`~/.nuget/packages/fractaldataworks.*`	Version-specific cache	NuGet global cache
`~/.nuget/packages/reference.*`	Version-specific cache	NuGet global cache

The forced restore (dotnet restore --force) in Phase 3 ensures consumer projects re-resolve all packages from the local folder rather than using cached copies.

4. pack-local.sh Internals

The scripts/pack-local.sh script is the standalone pack workflow for the FDW framework.

Usage:

./scripts/pack-local.sh                  # Build + pack (Release)
./scripts/pack-local.sh --no-build       # Pack only (reuse existing build output)
./scripts/pack-local.sh -c Debug         # Build + pack with Debug configuration

Key steps performed:

Cleans bin/, obj/, .vs/ directories across the entire repo
Detects version via nbgv get-version -v NuGetPackageVersion
Deletes old packages from LocalNugetFolder and ~/.nuget/packages cache
Creates a Fdw.Local.nuget.config in LocalNugetFolder with package source mapping:

<!-- Generated by pack-local.sh -->
<configuration>
  <packageSources>
    <clear />
    <add key="LocalFdw" value="%LocalNugetFolder%" />
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
  </packageSources>
  <packageSourceMapping>
    <packageSource key="LocalFdw">
      <package pattern="FractalDataWorks.*" />
      <package pattern="Reference.*" />
    </packageSource>
    <packageSource key="nuget.org">
      <package pattern="*" />
    </packageSource>
  </packageSourceMapping>
</configuration>

Builds and packs with dotnet pack --no-build
Generates Directory.Packages.props and FdwVersion.props in LocalNugetFolder
Updates versions.json in LocalNugetFolder for downstream version tracking

5. ReferenceSolutions/build-all.sh Internals

The ReferenceSolutions/build-all.sh script provides a more granular build workflow with selective solution targeting.

Usage:

./build-all.sh                          # Build everything (packs FDW first)
./build-all.sh --skip-fdw               # Skip FDW pack, just build solutions
./build-all.sh --skip-clean             # Skip clean step for faster iteration
./build-all.sh --solution Api           # Build only ApiSolution
./build-all.sh --solution Api,Etl       # Build multiple solutions
./build-all.sh --solution Shared        # Build only SharedPackages

Solution name mapping:

Short Name	Full Name
`Shared` / `SharedPackages`	SharedPackages
`Api` / `ApiSolution`	ApiSolution
`Etl` / `EtlServer`	EtlServer
`Scheduler` / `SchedulerServer`	SchedulerServer
`UI` / `ManagementUI`	ManagementUI

Build steps:

Step 1: Pack FractalDataWorks via pack-local.sh (unless --skip-fdw)
Step 1.5: Read versions.json from LocalNugetFolder and update FdwVersion in all Directory.Packages.props files
Step 2: Pack SharedPackages via pack-shared.sh
Steps 3-6: Build each solution using clean_and_build():

clean_and_build() {
    local project_path="$1" name="$2"
    dotnet clean "$project_path" -c Release-Local 2>/dev/null || true
    find "$project_dir" -type d -name "obj" -exec rm -rf {} + 2>/dev/null || true
    dotnet restore "$project_path" --force
    dotnet build "$project_path" -c Release-Local --no-restore
}

Note that ReferenceSolutions build with -c Release-Local, which uses the local NuGet source configuration.

6. start-service.sh

The ReferenceSolutions/start-service.sh script provides both an interactive TUI menu (using gum if available) and direct CLI commands for launching services.

CLI mode:

./start-service.sh api              # Start API (foreground)
./start-service.sh scheduler        # Start Scheduler
./start-service.sh etl              # Start ETL Server
./start-service.sh ui               # Start Blazor UI
./start-service.sh all              # Start all services (background)
./start-service.sh stop             # Stop all services
./start-service.sh status           # Show service status
./start-service.sh build            # Build all
./start-service.sh docker-reload    # Rebuild Docker containers
./start-service.sh db-reset         # Reset database in-place

Service definitions (from the script):

Key	Service	Port	Project Path
1	API	5001	`ApiSolution/src/Reference.Api/Reference.Api.csproj`
2	Scheduler	5005	`SchedulerServer/src/Reference.Scheduler.Server/Reference.Scheduler.Server.csproj`
3	ETL Server	5003	`EtlServer/src/Reference.Etl.Server/Reference.Etl.Server.csproj`
4	Management UI (Blazor)	5007	`ManagementUI/src/Reference.Management.UI/Reference.Management.UI.csproj`
5	Management UI (Tailwind)	5010	`ManagementUI-Tailwind/ManagementUI-Tailwind.csproj`
6	Management UI (WASM)	5011	`ManagementUI-WASM/src/Reference.Management.UI.Wasm.Server/...`

Environment secrets are set automatically by set_env_secrets(), which exports all FDW_SECRET_* variables with development defaults (see 17-02 Docker and Database Setup for the full list).

Dependency management: UI services require the API and optionally the Scheduler. The script checks dependencies before starting and can auto-start them.

7. Debug-Local Configuration

The -Local build configurations (Debug-Local, Develop-Local, Release-Local) enable local NuGet package resolution. They use the Fdw.Local.nuget.config file created by pack-local.sh to resolve FractalDataWorks.* and Reference.* packages from LocalNugetFolder instead of nuget.org.

When to use each configuration:

Configuration	Use Case
`Debug`	Framework-only development, no ReferenceSolutions
`Release`	Framework CI/CD, strict quality gates
`Debug-Local`	ReferenceSolutions with local framework packages, fast iteration
`Release-Local`	ReferenceSolutions with local framework packages, strict mode

8. Troubleshooting Build Failures

"LocalNugetFolder environment variable not set"

Run the setup script first:

./scripts/setup-local-nuget.sh
# Or manually:
export LocalNugetFolder="$HOME/development/local-nuget"

"Could not detect FDW version"

Ensure Nerdbank.GitVersioning is installed:

dotnet tool install -g nbgv

Stale package version errors

The NuGet cache may hold an old version. Clear it manually:

rm -rf ~/.nuget/packages/fractaldataworks.*
rm -rf ~/.nuget/packages/reference.*
rm -f "$LocalNugetFolder"/FractalDataWorks.*.nupkg
rm -f "$LocalNugetFolder"/Reference.*.nupkg

SharedPackages build fails after FDW version bump

The FdwVersion in SharedPackages/Directory.Packages.props may be out of sync. Run the full build-all.sh pipeline (which updates it automatically) or manually check cat "$LocalNugetFolder/versions.json" and update the props file.

ReferenceSolution restore fails with "package not found"

Verify: (1) LocalNugetFolder contains .nupkg files for the expected version, (2) Fdw.Local.nuget.config exists in LocalNugetFolder, (3) --force flag is used on restore.

Build succeeds but runtime fails

Usually a version mismatch between framework and SharedPackages. Run the full pipeline: cd ReferenceSolutions && ./build-all.sh.

17 01 Build Pipeline Guide

Build Pipeline Guide

Overview

1. build-all.sh Walkthrough (Framework-Level)

Phase 1: Build and Pack FDW Framework

Phase 2: Build and Pack SharedPackages

Phase 3: Build All ReferenceSolutions

2. Version Propagation

Version Detection

Version Flow

3. NuGet Cache Clearing

4. pack-local.sh Internals

5. ReferenceSolutions/build-all.sh Internals

6. start-service.sh

7. Debug-Local Configuration

8. Troubleshooting Build Failures

"LocalNugetFolder environment variable not set"

"Could not detect FDW version"

Stale package version errors

SharedPackages build fails after FDW version bump

ReferenceSolution restore fails with "package not found"

Build succeeds but runtime fails

Related Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!