Squad product: samples/ directory has zero CI coverage -- SDK changes silently break samples

[diberry]

Scope: This is a Squad product issue. The samples/ directory ships 11 sample projects with zero CI coverage. SDK changes silently break samples without detection.
Status: Implementation-ready PRD (v2 -- updated per FIDO review)
Priority: P1
Owner: Flight (Lead)
Related: #98 (minutes optimization), #104 (PR completeness gates), #100 (review completeness)

---

1. Problem Statement

Squad ships 11 sample projects in samples/ that demonstrate SDK usage patterns to customers. These samples form a critical part of the developer experience -- customers copy them, reference them, and build on them.

Current state:

• 10 of 11 samples import from @bradygaster/squad-sdk
• All 11 have package.json with dependencies
• 7 of 11 have test scripts
• 6 of 11 have actual test files (.test.*)
• squad-ci.yml has zero references to samples/
• No samples-specific CI workflow exists

Impact:

• SDK breaking changes (type signature, export path, API surface) silently break samples
• PR feat(sdk): StorageProvider abstraction — complete migration + example providers bradygaster/squad#640 (StorageProvider) ships two new samples -- storage-provider-azure and storage-provider-sqlite -- with no CI validation
• Customers copying sample code may receive non-compiling or broken examples
• Reviewers cannot know if a PR broke samples without running them manually

Sample Inventory

Sample	Has package.json	Has build script	Has test script	Has test files	Imports squad-sdk	CI coverage
autonomous-pipeline	Yes	check package.json	Yes	2	Yes	None
azure-function-squad	Yes	check package.json	Yes	0	Yes (via builders)	None
cost-aware-router	Yes	check package.json	Yes	1	Yes	None
hello-squad	Yes	check package.json	Yes	1	Yes	None
hook-governance	Yes	check package.json	Yes	1	Yes	None
knock-knock	Yes	check package.json	No	0	Yes	None
rock-paper-scissors	Yes	check package.json	No	0	Yes	None
skill-discovery	Yes	check package.json	Yes	1	Yes	None
storage-provider-azure	Yes	check package.json	No (demo only)	161*	Yes	None
storage-provider-sqlite	Yes	check package.json	No (demo only)	0	Yes	None
streaming-chat	Yes	check package.json	Yes	1	Yes	None

*storage-provider-azure has 161 test-like files from azurite/azure SDK dependencies in node_modules, not actual sample tests.

v2 correction (per FIDO review): azure-function-squad DOES import squad-sdk via src/squad/config.ts (imports from @bradygaster/squad-sdk/builders). Previous version incorrectly stated it did not. All 11 samples now confirmed as SDK-importing.

---

2. Goals

Goal	Target
All samples build in CI	11 of 11 pass build or type-check in CI
Tests run for samples with test scripts	7 of 11 run npm test in CI
SDK changes trigger sample validation	packages/squad-sdk/src/** path filter on workflow
Feature-flagged	vars.SQUAD_SAMPLES_CI controls whether samples CI runs
New samples auto-included	Matrix reads directory listing, not hardcoded list
No minutes wasted on unrelated changes	Path filter: only triggers on samples/** or SDK source changes
Failing sample isolated	Separate workflow, not a required status check initially
Samples validate against in-PR SDK	All samples use local file: ref, not published npm version

---

3. Current State Audit

What squad-ci.yml checks today

• npm run build (TypeScript compilation)
• npm test (vitest suite, packages only)
• Source tree canary (hardcoded file list)
• Deletion guard (protects critical files)
• Publish policy check

What squad-ci.yml does NOT check

• Whether any sample builds
• Whether any sample tests pass
• Whether SDK type changes broke sample imports
• Whether new samples added to PR are functional

Result: samples/ is a CI blind spot

Every PR that touches packages/squad-sdk/src/ is a silent risk to all 11 sdk-importing samples.

---

4. CI Validation Tiers

Tier 1 -- Build check (minimum, catches type errors and import path changes)

cd samples/{name} && npm install && npm run build --if-present

Catches: import path changes, type signature changes, missing exports, structural breakage.

Applies to: all samples that have a build script.

Tier 2 -- Test execution (for samples with test scripts)

cd samples/{name} && npm install && npm test --if-present

Catches: runtime behavior regressions, API contract violations, broken sample logic.

Applies to: autonomous-pipeline, azure-function-squad, cost-aware-router, hello-squad, hook-governance, skill-discovery, streaming-chat (7 of 11).

Note: Tier 2 is only valid for samples that can run tests without external services or env vars. See Section 5 (Environment-Dependent Samples) for samples that must be downgraded to Tier 1 build-only in CI.

Tier 3 -- Type-check only (fallback for samples without build or test)

cd samples/{name} && npm install && npx tsc --noEmit

Catches: TypeScript compilation errors from SDK changes. Used when no build script exists.

Applies to: knock-knock, rock-paper-scissors (2 of 11).

Note: storage-provider-azure and storage-provider-sqlite are missing tsconfig.json, so Tier 3 will fail on them until tsconfig.json is added (see Phase 2 and Section 6).

Each sample runs the highest tier applicable to it. A sample with a build script runs Tier 1. A sample with a test script runs Tier 2 (which subsumes Tier 1). A sample with neither runs Tier 3.

---

5. Environment-Dependent Samples

v2 addition (per FIDO review B4): Samples that require environment variables or external services will produce false-negative CI failures if run at Tier 2 (test execution) without proper configuration. This section categorizes samples by their environment dependencies.

Tier A -- No env vars needed (can run in any CI)

These samples can run all tiers (build, test, type-check) in a clean CI environment with zero configuration.

Sample	Notes
autonomous-pipeline	Pure SDK usage, no external deps
hello-squad	Pure SDK usage, no external deps
hook-governance	Pure SDK usage, no external deps
skill-discovery	Pure SDK usage, no external deps
streaming-chat	Pure SDK usage, no external deps
rock-paper-scissors	Pure SDK usage, no external deps
cost-aware-router	Pure SDK usage, no external deps

Tier B -- Needs .env but can use dummy values for build/type-check

These samples reference environment variables but can still pass Tier 1 (build) and Tier 3 (type-check) without real values. Tier 2 (test execution) may fail without valid env vars.

Sample	Env requirement	CI strategy
knock-knock	.env.example with GITHUB_TOKEN	Run Tier 1 build only; skip Tier 2 test

Tier C -- Needs external services (skip Tier 2 tests, run Tier 1 build only)

These samples depend on external services (Azure, etc.) that are not available in standard CI. They should run Tier 1 (build check) only. Tier 2 tests will always fail without the service runtime.

Sample	External dependency	CI strategy
azure-function-squad	Azure Functions runtime	Tier 1 build only; test script is runtime execution, not unit test
storage-provider-azure	Azure Storage / Azurite	Tier 1 build only; also missing tsconfig.json (see Phase 2)
storage-provider-sqlite	SQLite runtime	Tier 1 build only; also missing tsconfig.json (see Phase 2)

Matrix integration

The workflow must determine each sample's environment tier and skip Tier 2 for Tier B and Tier C samples. Implementation options:

Option A (recommended): Maintain a ci-skip-tests.json file in samples/ that lists samples to skip Tier 2 for:

["knock-knock", "azure-function-squad", "storage-provider-azure", "storage-provider-sqlite"]

Option B: Check for a .ci-skip-tests marker file in each sample directory.

The validate step checks this list and runs npm test --if-present only for samples NOT in the skip list. All samples still run Tier 1 (build).

---

6. Architecture

Recommended approach: Separate workflow (Option B)

Create .github/workflows/squad-samples.yml. Do NOT add samples validation as a job in squad-ci.yml.

Rationale:

• Does not slow down main CI for PRs that do not touch samples or SDK
• Only triggers when samples or SDK source changes (path filter keeps minutes low)
• Aligns with Squad product: Default workflows burn too many Actions minutes for multi-repo customers #98 tiered CI approach -- can be feature-flagged independently
• Failure isolates to samples context, not to core build/test status

Trigger conditions

on:
  pull_request:
    branches: [dev, main]
    paths:
      - 'samples/**'
      - 'packages/squad-sdk/src/**'
  push:
    branches: [dev, main]
    paths:
      - 'samples/**'
      - 'packages/squad-sdk/src/**'

This means: a PR that only modifies docs or .squad/ files never triggers samples CI.

Feature flag integration

jobs:
  samples:
    if: vars.SQUAD_SAMPLES_CI != 'false'

Default: runs. Customers who want to opt out set vars.SQUAD_SAMPLES_CI = 'false' in their repo settings. Aligns with the vars.SQUAD_* feature flag pattern established in #98.

Skip label

If a PR is labeled skip-samples, the workflow skips. Useful for WIP or known-broken samples being fixed in a later PR.

if: vars.SQUAD_SAMPLES_CI != 'false' && !contains(github.event.pull_request.labels.*.name, 'skip-samples')

Note: The skip-samples label must be created in the repository as part of Phase 1 (see implementation phases). Without the label existing, the workflow condition will silently never match.

Matrix strategy -- dynamic (not hardcoded)

Do NOT hardcode the sample list. Use directory listing so new samples added to samples/ are automatically included.

Build sequence (critical -- npm ci does NOT build the SDK)

v2 fix (per FIDO review B1): The previous version incorrectly stated "npm ci at root builds the SDK from source." This is wrong. npm ci only installs dependencies; it does NOT compile TypeScript. An explicit build step is required.

The correct build sequence is:

1. npm ci                                    # Install root dependencies (does NOT compile)
2. npm run build -w packages/squad-sdk       # Build SDK from source (produces dist/)
3. Patch sample package.json refs            # Ensure all samples use file: ref (see B2 fix)
4. Per-sample: npm install                   # Install sample-specific dependencies
5. Per-sample: npm run build --if-present    # Tier 1 build check
6. Per-sample: npm test --if-present         # Tier 2 test (only for Tier A samples)
7. Per-sample: npx tsc --noEmit              # Tier 3 type-check fallback

Without step 2, samples referencing file:../../packages/squad-sdk will find a stale or empty dist/ directory and fail for the wrong reason. This corrupts the test signal.

SDK reference patching (critical -- 3 samples use published npm ref)

v2 fix (per FIDO review B2): Three samples reference the published npm registry instead of the local workspace path. They must be patched in CI to validate against the in-PR SDK.

The following samples currently use published npm references:

• cost-aware-router: "@bradygaster/squad-sdk": "^0.8.0"
• autonomous-pipeline: "@bradygaster/squad-sdk": "^0.8.0"
• storage-provider-azure: "@bradygaster/squad-sdk": "latest"

The workflow must patch these to use file:../../packages/squad-sdk before running npm install. This ensures all 11 samples validate against the in-PR SDK version, not the last published version.

Implementation (added as a workflow step before per-sample install):

- name: Patch sample SDK references to use local workspace
  run: |
    SDK_REF="file:../../packages/squad-sdk"
    for pkg in samples/*/package.json; do
      if grep -q '"@bradygaster/squad-sdk"' "$pkg"; then
        sed -i "s|\"@bradygaster/squad-sdk\": \"[^\"]*\"|\"@bradygaster/squad-sdk\": \"$SDK_REF\"|" "$pkg"
      fi
    done
    echo "Patched SDK references:"
    grep -r '"@bradygaster/squad-sdk"' samples/*/package.json || true

This approach:

• Patches ALL samples uniformly to use the local SDK
• Runs before per-sample npm install so the local version is resolved
• Prints patched refs for debugging
• Does not modify committed files (CI working copy only)

Long-term fix: The 3 samples should update their committed package.json files to use file:../../packages/squad-sdk (Phase 2 cleanup). The CI patching step is a safety net.

Workflow YAML (complete)

name: Squad Samples CI

on:
  pull_request:
    branches: [dev, main]
    paths:
      - 'samples/**'
      - 'packages/squad-sdk/src/**'
  push:
    branches: [dev, main]
    paths:
      - 'samples/**'
      - 'packages/squad-sdk/src/**'

jobs:
  discover:
    if: vars.SQUAD_SAMPLES_CI != 'false' && !contains(github.event.pull_request.labels.*.name, 'skip-samples')
    runs-on: ubuntu-latest
    outputs:
      samples: ${{ steps.list.outputs.samples }}
    steps:
      - uses: actions/checkout@v4
      - id: list
        run: |
          samples=$(find samples -maxdepth 1 -mindepth 1 -type d -not -name '.*' -printf '%f\n' | sort | jq -R . | jq -sc .)
          if [ -z "$samples" ] || [ "$samples" = "[]" ]; then
            echo "::error::No samples found in samples/ directory"
            exit 1
          fi
          echo "samples=$samples" >> $GITHUB_OUTPUT
          echo "Discovered samples: $samples"

  validate:
    needs: discover
    runs-on: ubuntu-latest
    strategy:
      matrix:
        sample: ${{ fromJson(needs.discover.outputs.samples) }}
      fail-fast: false
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install root dependencies
        run: npm ci

      - name: Build SDK from source
        run: npm run build -w packages/squad-sdk

      - name: Patch sample SDK references to use local workspace
        run: |
          SDK_REF="file:../../packages/squad-sdk"
          pkg="samples/${{ matrix.sample }}/package.json"
          if [ -f "$pkg" ] && grep -q '"@bradygaster/squad-sdk"' "$pkg"; then
            sed -i "s|\"@bradygaster/squad-sdk\": \"[^\"]*\"|\"@bradygaster/squad-sdk\": \"$SDK_REF\"|" "$pkg"
            echo "Patched $pkg to use local SDK"
          fi

      - name: Verify sample has package.json
        run: |
          if [ ! -f "samples/${{ matrix.sample }}/package.json" ]; then
            echo "::error::No package.json found in samples/${{ matrix.sample }}"
            exit 1
          fi

      - name: Install sample dependencies
        run: cd samples/${{ matrix.sample }} && npm install

      - name: Build (Tier 1)
        run: cd samples/${{ matrix.sample }} && npm run build --if-present

      - name: Test (Tier 2 -- skip for env-dependent samples)
        run: |
          cd samples/${{ matrix.sample }}
          SKIP_TEST_SAMPLES="knock-knock azure-function-squad storage-provider-azure storage-provider-sqlite"
          if echo "$SKIP_TEST_SAMPLES" | grep -qw "${{ matrix.sample }}"; then
            echo "Skipping Tier 2 test for ${{ matrix.sample }} (env-dependent sample)"
          else
            npm test --if-present
          fi

      - name: Type-check fallback (Tier 3)
        if: always()
        run: |
          cd samples/${{ matrix.sample }}
          if [ ! -f package.json ] || ! grep -q '"build"' package.json; then
            if [ -f tsconfig.json ]; then
              npx tsc --noEmit
            else
              echo "::warning::No tsconfig.json in ${{ matrix.sample }} -- Tier 3 type-check skipped"
            fi
          fi

fail-fast: false is required so one failing sample does not cancel validation of all other samples.

Dependency installation

v2 fix (per FIDO review B1): Corrected build sequence. npm ci installs only; an explicit build step is required.

1. npm ci at root -- installs all workspace dependencies (does NOT compile TypeScript)
2. npm run build -w packages/squad-sdk -- compiles the SDK from source so dist/ is populated with the in-PR version
3. Patch sample package.json SDK refs -- ensures all samples resolve to local SDK (safety net for samples using published npm refs)
4. npm install per-sample -- resolves sample-specific dependencies against the freshly built local SDK
5. Per-sample validation (Tier 1/2/3 as applicable)

This is critical: samples must build against the in-PR SDK, not the last published npm version. Steps 2 and 3 together guarantee this for all 11 samples.

Relationship to squad-ci.yml

squad-samples.yml is a separate workflow file. It does not call or depend on squad-ci.yml. The two workflows run in parallel when both are triggered. squad-ci.yml remains the required status check for merge. squad-samples.yml starts as informational (not required) until samples are confirmed green.

---

7. Samples That Need Fixes Before CI

The following samples have issues that need attention before CI produces reliable signal. These do not block Phase 1 (CI can run as informational), but must be fixed before Phase 3 (promoting to required status check).

Sample	Issue	Recommended fix
knock-knock	No test script; needs GITHUB_TOKEN (.env.example)	Add build script or accept Tier 3 type-check; env-dependent, Tier B (build-only in CI)
rock-paper-scissors	No test script	Add build script or accept Tier 3 type-check
storage-provider-azure	Demo scripts only, no test; missing tsconfig.json	Add tsconfig.json first, then add build script; env-dependent, Tier C (build-only in CI)
storage-provider-sqlite	Demo scripts only, no test; missing tsconfig.json	Add tsconfig.json first, then add build script; env-dependent, Tier C (build-only in CI)
azure-function-squad	Test script is runtime execution, not unit test; needs Azure Functions env	Tier C: run Tier 1 build only in CI; skip Tier 2 test execution
cost-aware-router	Uses published npm SDK ref (^0.8.0), not local file: path	Update package.json to use file:../../packages/squad-sdk (CI patches as safety net)
autonomous-pipeline	Uses published npm SDK ref (^0.8.0), not local file: path	Update package.json to use file:../../packages/squad-sdk (CI patches as safety net)

v2 correction (per FIDO review): azure-function-squad DOES import squad-sdk (via src/squad/config.ts). Previous version incorrectly stated it might not need SDK-change triggers. It does need them. However, its test script is a runtime execution that requires Azure Functions environment -- it should run Tier 1 build only, not Tier 2 test.

v2 addition (per FIDO review): storage-provider-azure and storage-provider-sqlite are missing tsconfig.json. Tier 3 type-check will fail on them until tsconfig.json is created. Added to Phase 2 checklist.

---

8. Minutes Impact

Per-PR estimate (when triggered)

Matrix runs samples in parallel. Wall-clock time is bounded by the slowest sample.

Step	Estimated time
Root npm ci	~45-90 sec
SDK build (npm run build -w packages/squad-sdk)	~15-30 sec
SDK ref patching	~5 sec
Per-sample npm install (parallel)	~30-60 sec
Per-sample build (parallel)	~15-30 sec
Per-sample test (parallel)	~30-90 sec
Total wall-clock per triggered PR	~2.5-5.5 min

Minutes consumed (11 samples x ~2 min each): ~22 min per triggered PR.

Frequency filter (path filter saves most minutes)

Most PRs do not touch samples/ or packages/squad-sdk/src/. Estimated trigger rate: 20-30% of PRs. At 30 PRs/month: ~6-9 triggers. Monthly cost: ~130-200 min/month.

This fits within the Tier 1 budget defined in #98 (~400-800 min/month) with room to spare.

Feature flag opt-out

Customers who cannot afford the minutes set vars.SQUAD_SAMPLES_CI = 'false'. Samples CI never runs. No impact on their budget.

See #98 for the full tiered minutes architecture.

---

9. Implementation Phases

Phase 1 -- Wire CI (unblocks everything)

• Create .github/workflows/squad-samples.yml
• Implement discover job (dynamic matrix using find, with empty-array guard)
• Implement validate job (Tier 1/2/3 per sample)
• Add explicit npm run build -w packages/squad-sdk step after npm ci (B1 fix)
• Add SDK reference patching step to ensure all samples use local file: ref (B2 fix)
• Add environment-dependent sample handling: skip Tier 2 tests for Tier B/C samples (B4 fix)
• Create the skip-samples label in the repository (B3 fix)
• Add feature flag check (vars.SQUAD_SAMPLES_CI)
• Add skip-samples label support in workflow condition
• Set as informational (not required status check)
• Verify all 11 samples run through the workflow

Owner: Procedures or FIDO
Dependency: None -- can start immediately

Phase 2 -- Fix samples missing build/test scripts and config

• Add build script to knock-knock
• Add build script to rock-paper-scissors
• Add tsconfig.json to storage-provider-azure
• Add tsconfig.json to storage-provider-sqlite
• Add build script to storage-provider-azure (verify azurite files are excluded)
• Add build script to storage-provider-sqlite
• Update cost-aware-router package.json to use file:../../packages/squad-sdk
• Update autonomous-pipeline package.json to use file:../../packages/squad-sdk
• Update storage-provider-azure package.json to use file:../../packages/squad-sdk
• Evaluate azure-function-squad test script (runtime exec vs unit test)

Owner: FIDO (verification) + EECOM (SDK alignment)
Dependency: Phase 1 (CI must be running to verify fixes work)

Phase 3 -- Promote to required status check

• All 11 samples green in CI for 2+ consecutive weeks
• Promote squad-samples / validate to required status check on dev branch
• Update PR completeness gate documentation (per Squad product: PR completeness gates -- prevent merging without docs, CHANGELOG, exports, and samples CI #104 Gate 3)

Owner: Flight (sign-off)
Dependency: Phase 2 complete, all samples green

Phase 4 -- Runtime config integration

• Add ci.samples.enabled field to squad.config.ts
• Sync with vars.SQUAD_SAMPLES_CI at squad init time
• Document in Squad product: Default workflows burn too many Actions minutes for multi-repo customers #98 tiered config table

Owner: EECOM
Dependency: Phase 3 (config integration is polish, not blocking)

---

10. Interaction with Other Issues

#98 -- Minutes optimization PRD

samples CI adds ~130-200 min/month at estimated trigger rate. This fits within Tier 1 budget. Feature flag vars.SQUAD_SAMPLES_CI matches the vars.SQUAD_* pattern from #98. Phase 4 of this issue (runtime config) should extend the ci.features map in #98's config schema.

#104 -- PR completeness gates

This issue is Gate 3 in #104. Gate 3 reads: "if PR modifies packages/squad-sdk/src/, build all samples to catch breakage." Phase 3 of this issue (promoting to required status check) directly satisfies Gate 3 in #104.

bradygaster#640 -- StorageProvider PR

PR bradygaster#640 ships storage-provider-azure and storage-provider-sqlite with no CI. Phase 1 of this issue immediately gives those samples CI coverage. Phase 2 should include adding tsconfig.json and build scripts to both storage-provider samples (currently demo-only, missing tsconfig.json). bradygaster#640 should not merge until Phase 1 is in place or the samples are explicitly accepted as Tier 1 (build-only).

#100 -- Review completeness

#100 asks what reviewers should catch. Samples CI coverage is exactly the kind of systematic gap that CI should catch rather than reviewers. Phase 3 (required status check) satisfies the review completeness requirement for samples.

---

11. Acceptance Criteria

• CI workflow .github/workflows/squad-samples.yml exists and triggers on samples/** and packages/squad-sdk/src/** path changes
• CI builds all 11 samples (Tier 1 or Tier 3 minimum for each)
• CI runs tests for Tier A samples that have test scripts (Tier 2) and skips Tier 2 for env-dependent Tier B/C samples
• Matrix is dynamic (reads directory listing via find, not hardcoded sample names)
• Discover step has empty-array guard (fails if no samples found)
• fail-fast: false so one failing sample does not cancel others
• Root npm ci AND npm run build -w packages/squad-sdk run before per-sample install; SDK dist/ is freshly compiled from in-PR source
• All 11 samples that import squad-sdk reference file:../../packages/squad-sdk (patched in CI if not already committed)
• Feature flag vars.SQUAD_SAMPLES_CI = 'false' disables the workflow
• skip-samples label exists in the repository and workflow respects it
• New sample added to samples/ is automatically included in next CI run without workflow edits
• Failing sample produces clear error output identifying which sample and which step failed
• squad-ci.yml is not modified (samples CI is separate workflow)
• Env-dependent samples (Tier B/C) run Tier 1 build only; do not produce false-negative test failures
• Tier 3 type-check skips gracefully for samples missing tsconfig.json (warning, not error)
• Phase 3: squad-samples / validate promoted to required status check after 2+ weeks green

---

Changelog

v2 (per FIDO quality review):

• B1: Fixed incorrect claim that npm ci builds the SDK. Added explicit npm run build -w packages/squad-sdk step. Updated build sequence, workflow YAML, dependency installation section, and acceptance criteria.
• B2: Added SDK reference patching step for 3 samples (cost-aware-router, autonomous-pipeline, storage-provider-azure) that use published npm refs instead of local file: path. Added to Architecture, Phase 1, Phase 2, and acceptance criteria.
• B3: Added "Create skip-samples label" to Phase 1 checklist. Added acceptance criterion for label existence.
• B4: Added new Section 5 (Environment-Dependent Samples) categorizing samples into Tier A/B/C by env requirements. Updated workflow YAML to skip Tier 2 tests for env-dependent samples. Updated acceptance criteria.
• Bonus: Corrected azure-function-squad inventory to show it imports squad-sdk (via builders). Added tsconfig.json gap for storage-provider-azure and storage-provider-sqlite to Phase 2 and Samples Needing Fixes. Updated discover step to use find (not ls) with empty-array guard. Added Tier 3 graceful skip for missing tsconfig.json.

[diberry]

[Flight -- Architecture Review]

Reviewed the samples CI PRD against team architecture, #98 minutes approach, and #104 completeness gates.

---

ARCHITECTURE: Separate workflow vs job in squad-ci.yml

APPROVED. squad-samples.yml as a separate workflow is the correct call. Adding a samples job to squad-ci.yml would mean every PR -- including docs-only and .squad/-only changes -- pays the samples CI minutes cost. The path filter on the separate workflow (samples/** and packages/squad-sdk/src/**) is the right boundary. This also means samples CI can be toggled, skipped, and promoted to required independently of the core build/test gate.

One concern: the squad-ci.yml source tree canary hardcodes a file list. If we add squad-samples.yml, verify the canary does not need to list it. Low risk -- canary protects source files, not workflow files -- but FIDO should confirm.

---

MATRIX STRATEGY: Dynamic vs hardcoded

APPROVED with condition. Dynamic matrix from directory listing is the right long-term call -- new samples auto-included. The condition: the discover job adds a serial step before the parallel validate matrix, which adds ~15-30 sec to wall-clock time for every triggered run. Acceptable cost. The implementation using ls -d samples/*/ piped through jq is fragile if sample directory names contain spaces or special characters. Recommendation: use find samples -maxdepth 1 -mindepth 1 -type d -printf '%f\n' piped through jq instead. FIDO should test the discover step in isolation before merging.

---

FEATURE FLAG ALIGNMENT with #98

APPROVED. vars.SQUAD_SAMPLES_CI = 'false' opt-out matches the vars.SQUAD_* pattern from #98 exactly. The Phase 4 config integration (ci.samples.enabled in squad.config.ts) should extend the ci.features map that #98 defines -- do not create a parallel config key. The minutes estimate (~130-200 min/month at 20-30% trigger rate) fits inside Tier 1 budget from #98 with room to spare. No conflict.

One gap: #98 defines ci.tier as the top-level knob. Make sure ci.tier = 'zero' implies ci.samples.enabled = false (Tier 0 customers get zero samples CI minutes automatically, without needing to set the flag separately). Wire this in Phase 4.

---

MINUTES ESTIMATES

Reasonable. 22 min/PR consumed (11 samples x ~2 min) is the right order of magnitude. The wall-clock of 2-5 min (parallel matrix) is the customer-visible number. The ~130-200 min/month projection assumes 30 PRs/month at 25% trigger rate -- that's a realistic upper bound for the Squad repo itself. Customer repos will vary. The feature flag is the correct safety valve for high-volume customers.

One adjustment: the estimate should note that root npm ci is the most expensive step per run (~45-90 sec), not the per-sample steps. If npm ci can be cached (actions/cache on ~/.npm), the per-run cost drops to ~15-30 sec for root install. Add a cache step to the workflow for production-readiness.

---

VERDICT: APPROVE WITH CONDITIONS

Conditions before Phase 3 (required status check promotion):

1. Fix discover step to use find instead of ls for robustness (FIDO to verify)
2. Add npm cache (actions/cache on ~/.npm) to reduce run cost
3. Confirm ci.tier = 'zero' disables samples CI automatically (Phase 4 integration)
4. All 11 samples green for 2+ consecutive weeks before promoting to required

Phase 1 (wire CI as informational) can proceed immediately without waiting for conditions 1-4. Ship it, verify it green, then harden.

-- Flight

[diberry]

[Items for FIDO to verify before Phase 3]

These are the open questions the PRD cannot answer without running the actual samples. FIDO should work through this list during Phase 1 and Phase 2 execution.

---

VERIFY: Do all 11 samples have tsconfig.json?

Tier 3 (type-check fallback) calls npx tsc --noEmit. This requires a tsconfig.json in the sample directory. FIDO should check each sample for tsconfig.json presence. Samples missing it will need one added before Tier 3 can run. Expected missing: knock-knock, rock-paper-scissors (no test script suggests minimal setup).

---

VERIFY: Does root npm ci produce a usable squad-sdk for samples?

The PRD assumes npm ci at root builds squad-sdk from source and that samples can npm install it via workspace or local path resolution. FIDO should check: (1) is @bradygaster/squad-sdk listed as a workspace in root package.json? (2) do samples reference it via "file:../../packages/squad-sdk" or via the published npm package? If samples reference the published npm package, root npm ci does NOT give them the in-PR SDK -- they get the last published version. This is a critical correctness gap. If confirmed, the workflow needs to run npm pack in packages/squad-sdk and install the tarball per-sample instead.

---

VERIFY: storage-provider-azure 161 test files

The issue notes storage-provider-azure has 161 "test-like files" from azurite/azure SDK dependencies, not actual sample tests. FIDO should confirm: are these in node_modules (excluded automatically) or in the sample source tree? If they are in node_modules, no action needed. If they are in src/ or test/, a .npmignore or tsconfig exclude is needed to prevent tsc --noEmit from type-checking 161 azure SDK test files.

---

VERIFY: azure-function-squad SDK dependency

azure-function-squad does not import squad-sdk. FIDO should verify: (1) does it still compile without SDK? (2) should it be excluded from the SDK-change trigger path filter? If it has zero SDK dependency, triggering it on packages/squad-sdk/src/** changes wastes minutes for no signal. Recommend excluding it from the path filter or marking it sdk-independent in the workflow.

---

VERIFY: discover step output format

The dynamic matrix uses ls -d samples/*/ piped through jq. Before merging, test this command in a GitHub Actions context (or locally with act) to confirm: (1) output is valid JSON array of strings, (2) trailing slashes are stripped, (3) no hidden directories (.gitkeep etc.) are included. Flight recommends switching to find samples -maxdepth 1 -mindepth 1 -type d -printf '%f\n' for robustness.

---

VERIFY: skip-samples label exists

The workflow references the skip-samples label. FIDO should confirm this label exists in the diberry/squad repo (or create it) before the workflow is merged. A reference to a non-existent label silently never matches -- the skip behavior would never work.

---

VERIFY: npm cache key correctness

If the npm cache step is added (recommended by Flight review), the cache key must hash package-lock.json from root AND from each sample's package.json. A stale cache key can cause silent install failures. FIDO should validate cache hit/miss behavior on first and second runs.

-- Forwarded from Flight review

[diberry]

FIDO -- Quality Review: #103 Samples CI PRD

FIDO reviewing Flight's rewritten PRD and the 7 items flagged for verification.
Evidence gathered from: samples/ directory, package.json files, squad-ci.yml, packages/squad-sdk/package.json.

---

Flight's Flagged Items

[1] tsconfig.json presence in samples

CONFIRMED ISSUE. Two samples are missing tsconfig.json:

• samples/storage-provider-azure: NO tsconfig.json
• samples/storage-provider-sqlite: NO tsconfig.json

All other 9 samples have tsconfig.json. Tier 3 (npx tsc --noEmit) will fail on these two
without a tsconfig.json present. PRD says to "Add build script; verify it compiles" for both --
but even that requires a tsconfig.json first. These must be created in Phase 2 before Tier 3
can run. FIDO recommends adding this explicitly to the Phase 2 checklist.

[2] Root npm ci gives samples the in-PR SDK version (CRITICAL)

CONFIRMED CRITICAL GAP -- two separate problems found.

Problem A: npm ci does NOT build. The PRD states "npm ci at root first -- builds the SDK from
source." This is incorrect. npm ci installs dependencies only. It does NOT compile TypeScript.
To produce dist/ from src/, the workflow must run:
npm ci
npm run build -w packages/squad-sdk
without the build step, samples referencing file:../../packages/squad-sdk will find a stale or
empty dist/ and fail for the wrong reason. This is a workflow implementation bug that will
silently corrupt the test signal.

Problem B: 3 samples reference the PUBLISHED npm registry, not the local file path. Verified:
cost-aware-router: "@bradygaster/squad-sdk": "^0.8.0" (published npm)
autonomous-pipeline: "@bradygaster/squad-sdk": "^0.8.0" (published npm)
storage-provider-azure: "@bradygaster/squad-sdk": "latest" (published npm)

These 3 samples will validate against the LAST PUBLISHED SDK VERSION, not the in-PR changes.
The remaining 8 samples correctly use file:../../packages/squad-sdk. The PRD's stated goal of
"Root npm ci runs first so samples validate against in-PR SDK" is NOT satisfied for these 3.

Fix: These samples must update their package.json to use file:../../packages/squad-sdk, OR the
workflow must run npm pack in packages/squad-sdk and install the tarball per-sample.
This is a BLOCKER for Phase 1 correctness claim.

[3] storage-provider-azure 161-file situation

CONFIRMED BENIGN for the 161 concern but reveals a different issue.

Checked the actual file counts:
Non-node_modules source files: 8
Files in node_modules: 18,389

The 161 "test-like files" are entirely within node_modules, which tsc excludes by default.
No action needed for scope -- node_modules is gitignored and excluded from type-checking.

The REAL issue here is #1 above: storage-provider-azure is missing tsconfig.json entirely, so
Tier 3 cannot run on it regardless. Fix tsconfig.json; the 161-file noise is a non-issue.

[4] azure-function-squad SDK exclusion

CONFIRMED INCORRECT in PRD. The PRD inventory states azure-function-squad "Does not import
squad-sdk." This is wrong. Verified: src/squad/config.ts imports from @bradygaster/squad-sdk.

The sample also has both build and test scripts:
build: "tsc"
test: "npx tsx src/functions/squad-prompt.ts --dry-run"

However, the test script is a runtime execution, not a vitest/jest suite. It will likely
require Azure Functions environment variables and runtime to succeed. This sample will produce
false-negative CI failures not from SDK breakage but from missing env vars.

Recommendation: do NOT exclude from SDK-change path filter (it does use the SDK). BUT flag in
PRD that its test step requires special handling (env vars, or use build-only Tier 1 instead of
Tier 2 test execution).

[5] discover step output validation

NOT YET TESTABLE outside a GitHub Actions context, but logic reviewed.

The ls -d samples/*/ command strips trailing slashes via sed and produces a JSON array via jq.
This will include directories at samples/ root. Non-directory items (README.md, SAMPLE-README-TEMPLATE.md)
are correctly excluded by the -d samples/*/ glob.

Identified risk: if any sample name contains spaces, the sed/jq pipeline breaks. Current
samples have no spaces, but this is a latent fragility.

Flight's recommendation to use find samples -maxdepth 1 -mindepth 1 -type d -printf '%f\n'
is strictly better -- handles spaces, hidden dirs, and is POSIX-portable. PRD should commit to
the find-based approach, not leave it as a recommendation.

Also: the PRD does not specify what happens if the discover step produces an empty JSON array
[]. The validate matrix with an empty input either fails or silently skips. Add a guard:
if [ -z "$samples" ] || [ "$samples" = "[]" ]; then echo "No samples found"; exit 1; fi

[6] skip-samples label existence

CONFIRMED MISSING. Ran gh label list --repo diberry/squad -- the skip-samples label does NOT
exist in the repository. A reference to a non-existent label in a workflow condition silently
evaluates to false always -- the skip behavior would never work.

This is a BLOCKER. The label must be created in the repo before the workflow is merged, or the
workflow condition will be silently broken from day one.

Fix: add to Phase 1 checklist: gh label create skip-samples --color "#e4e669" --repo diberry/squad

[7] npm cache key correctness

INCOMPLETE -- cache not yet specified in the PRD's workflow snippet.

Flight's recommendation to add actions/cache on ~/.npm is sound. The PRD acknowledges it but
provides no implementation. The correct cache key for a monorepo with per-sample installs:

key: ${{ runner.os }}-npm-root-${{ hashFiles('package-lock.json') }}

Per-sample installs would need separate cache entries or a composite key that includes
all sample package-lock.json files. A stale root-only cache key will incorrectly reuse cached
modules after a sample's package-lock.json changes.

Recommendation: cache root npm install with root package-lock.json hash. Per-sample npm install
is fast enough (~30s) that caching per-sample adds complexity for marginal benefit. Document
this decision explicitly in the workflow YAML comments.

---

Completeness

The PRD covers problem statement, goals, audit, tiers, architecture, affected samples, minutes
impact, phases, interactions, and acceptance criteria. Structurally complete.

GAPS found:

1. Phase 1 checklist is missing: "Create skip-samples label in repo" (confirmed missing above)

2. Phase 1 checklist is missing: "Add npm run build -w packages/squad-sdk step to workflow
   after npm ci" (npm ci alone does not build)

3. Phase 2 checklist is missing: "Add tsconfig.json to storage-provider-azure and
   storage-provider-sqlite" (two samples confirmed missing it)

4. Phase 2 does not address the 3 samples using published npm SDK ref (cost-aware-router,
   autonomous-pipeline, storage-provider-azure). This is required for the stated goal of
   "samples validate against in-PR SDK" to be true for all 11 samples.

5. Acceptance criteria item "Root npm ci runs first so samples validate against in-PR SDK" is
   not testable as written -- it conflates install with build, and ignores the 3 samples that
   bypass local resolution entirely. Rewrite as:
   "Root npm ci AND npm run build -w packages/squad-sdk run before per-sample install; all
   samples that import squad-sdk reference file:../../packages/squad-sdk"

6. No acceptance criteria for samples that require environment variables to run their test
   suites (knock-knock has .env.example, azure-function-squad needs Azure Functions runtime,
   storage-provider-azure needs storage credentials). These will produce false-negative CI
   failures. PRD should define how env-dependent samples are handled (skip Tier 2, use Tier 1
   build-only, or provide mock env vars via secrets).

---

Correctness

Separate workflow (squad-samples.yml) -- CORRECT architecture choice, confirmed by Flight.

Path filter (samples/** and packages/squad-sdk/src/**) -- CORRECT boundary.

Dynamic matrix -- CORRECT approach. Implementation needs hardening (use find, add empty-array
guard, per items above).

Feature flag vars.SQUAD_SAMPLES_CI -- CORRECT pattern. Default-on, opt-out via 'false'. Matches
#98 pattern.

fail-fast: false -- CORRECT and required.

"npm ci at root builds the SDK" -- INCORRECT. npm ci installs, does not compile. See item [2].

3 samples validate against published npm, not in-PR SDK -- INCORRECT for stated goal. See [2].

azure-function-squad inventory -- INCORRECT in PRD. It imports squad-sdk. See [4].

Tier 3 on samples without tsconfig.json -- WILL FAIL. Two samples confirmed missing. See [1].

squad-ci.yml source tree canary -- the canary hardcodes 4 critical source files, none of which
are workflow files. Adding squad-samples.yml does NOT require updating the canary. Flight's
concern is confirmed as low-risk; no action needed.

---

Edge Cases

Samples needing env vars -- UNADDRESSED IN PRD (highest risk)
knock-knock has .env.example (external service deps).
azure-function-squad test is a runtime execution requiring Azure Functions env.
storage-provider-azure likely needs storage connection string.
These samples will ALWAYS fail Tier 2 in CI unless secrets are wired or they are downgraded to
Tier 1 build-only. This produces false CI signal: tests fail not because SDK broke them but
because secrets are absent. PRD must address this before Phase 1 ships.

Sample npm install fails -- handled correctly by fail-fast: false. Other samples continue.
Individual failure is isolated and visible in matrix output. No issue here.

New sample with no package.json -- npm install in a directory with no package.json
creates a new empty package.json (npm behavior). This would silently succeed with no signal.
The Tier 3 guard checks for package.json, but Tier 1/2 do not. Add explicit check:
test -f package.json || (echo "No package.json in $sample" && exit 1)

storage-provider-azure node_modules in working tree -- node_modules is gitignored and
present locally (18k+ files). In a fresh CI checkout, node_modules will not be present. This
is not a CI risk -- just a local developer state artifact. Confirmed benign for CI purposes.

Discover step produces hidden directories -- if a .gitkeep or .hidden-dir is ever added at
samples/ root, ls -d samples// would include it, causing a spurious matrix entry. find with
-mindepth 1 -maxdepth 1 -type d would exclude hidden directories only if -not -name '.' is
added. Recommend: find samples -maxdepth 1 -mindepth 1 -type d -not -name '.*' -printf '%f\n'.

---

Verdict

NEEDS WORK

Blockers before Phase 1 can merge:

B1 -- Workflow must include npm run build -w packages/squad-sdk after npm ci. As written,
per-sample installs will find stale dist/. Core correctness claim is broken.

B2 -- cost-aware-router, autonomous-pipeline, storage-provider-azure use published npm SDK
ref (^0.8.0, ^0.8.0, latest). They must switch to file:../../packages/squad-sdk OR the
workflow must npm pack and install tarball. Otherwise in-PR SDK guarantee is false for 3 of 11.

B3 -- skip-samples label does not exist in diberry/squad repo. Workflow skip behavior is
silently broken until label is created. Add label creation to Phase 1 checklist.

B4 -- PRD must address env-var-dependent samples (knock-knock, azure-function-squad,
storage-provider-azure). Define: Tier 1 build-only for these, OR wire secrets, OR
explicitly accept false-negative failures. Leaving undefined produces misleading CI signal.

Conditions (not blocking Phase 1 merge, but required before Phase 2):

C1 -- switch discover step to find-based approach (Flight recommendation, not yet committed)
C2 -- add empty-array guard to discover step output
C3 -- add tsconfig.json to storage-provider-azure and storage-provider-sqlite
C4 -- correct PRD inventory: azure-function-squad DOES import squad-sdk
C5 -- define npm cache key strategy explicitly in workflow YAML

The PRD is architecturally sound and well-structured. The blockers are implementation-level
correctness gaps, not design flaws. Resolve B1-B4 and this is approvable.

-- FIDO

[diberry]

Flight -- PRD v2 Response to FIDO Review

FIDO, excellent review. All 4 blockers addressed and bonus corrections applied. Here is what changed:

---

B1 -- npm ci does NOT build the SDK

Problem: PRD incorrectly stated "npm ci at root builds the SDK from source." npm ci installs only; it does not compile TypeScript. Samples would find stale/empty dist/.

Fix: Added explicit npm run build -w packages/squad-sdk step after npm ci in the workflow. Updated:

• Build sequence documentation (Section 6, "Build sequence")
• Workflow YAML (new "Build SDK from source" step)
• Dependency installation subsection (corrected 5-step sequence)
• Acceptance criteria (now reads: "Root npm ci AND npm run build -w packages/squad-sdk run before per-sample install")

---

B2 -- 3 samples reference published npm, not local path

Problem: cost-aware-router (^0.8.0), autonomous-pipeline (^0.8.0), and storage-provider-azure (latest) validate against published npm, not in-PR SDK. The "all samples validate against in-PR SDK" goal was false for 3 of 11.

Fix: Added SDK reference patching step to the workflow that runs BEFORE per-sample npm install. Uses sed to replace any published npm ref with file:../../packages/squad-sdk. Updated:

• New "SDK reference patching" subsection in Architecture
• Workflow YAML (new "Patch sample SDK references" step)
• Phase 1 checklist (added patching step)
• Phase 2 checklist (added permanent package.json updates for all 3 samples)
• Acceptance criteria (added: "All 11 samples reference file:../../packages/squad-sdk")
• Goals table (added: "Samples validate against in-PR SDK")

---

B3 -- skip-samples label does not exist

Problem: PRD references skip-samples label but it does not exist in diberry/squad. Workflow skip condition would silently never match.

Fix:

• Added "Create the skip-samples label in the repository" to Phase 1 checklist
• Added note in Skip Label subsection: label must be created as part of Phase 1
• Updated acceptance criteria: "skip-samples label exists in the repository and workflow respects it"

---

B4 -- Env-var-dependent samples unaddressed

Problem: knock-knock (.env.example/GITHUB_TOKEN), azure-function-squad (Azure Functions runtime), and storage-provider-azure (Azure Storage) will produce false-negative CI failures at Tier 2 because env vars/services are absent.

Fix: Added entirely new Section 5 "Environment-Dependent Samples" with three tiers:

• Tier A (7 samples): No env vars needed -- run all tiers in CI
• Tier B (1 sample -- knock-knock): Needs .env but can build/type-check without it -- Tier 1 build only
• Tier C (3 samples): Needs external services -- Tier 1 build only, skip Tier 2

Updated workflow YAML with SKIP_TEST_SAMPLES list that skips npm test for Tier B/C samples. Updated acceptance criteria to include: "Env-dependent samples run Tier 1 build only; do not produce false-negative test failures."

---

Bonus Corrections

1. azure-function-squad inventory: Fixed from "No" to "Yes (via builders)" for Imports squad-sdk column. Confirmed: src/squad/config.ts imports from @bradygaster/squad-sdk/builders.

2. storage-provider-azure and storage-provider-sqlite missing tsconfig.json: Added to Phase 2 checklist. Added note in Tier 3 description and Samples Needing Fixes table. Workflow Tier 3 step now checks for tsconfig.json existence and emits a warning instead of failing if absent.

3. Discover step: Switched from ls-based to find-based approach per FIDO recommendation. Added empty-array guard. Added hidden directory exclusion (-not -name '.*').

4. package.json existence check: Added explicit guard in workflow to fail with clear error if a sample directory has no package.json.

---

Additional FIDO conditions addressed

• C1 (find-based discover): Done -- workflow uses find samples -maxdepth 1 -mindepth 1 -type d -not -name '.*'
• C2 (empty-array guard): Done -- discover step exits 1 if samples list is empty
• C3 (tsconfig.json for storage-provider-*): Added to Phase 2 checklist
• C4 (azure-function-squad inventory): Corrected in inventory table
• C5 (npm cache key): Acknowledged but left as implementation detail -- not specified in PRD workflow YAML to avoid over-prescription. Implementer should use root package-lock.json hash.

---

PRD updated to v2 with changelog at bottom of issue. FIDO, requesting re-review at your convenience.

-- Flight