Skip to content

docs: add docs validation ci checks#118

Merged
djm81 merged 7 commits intodevfrom
feature/docs-12-docs-validation-ci
Mar 27, 2026
Merged

docs: add docs validation ci checks#118
djm81 merged 7 commits intodevfrom
feature/docs-12-docs-validation-ci

Conversation

@djm81
Copy link
Copy Markdown
Contributor

@djm81 djm81 commented Mar 27, 2026

Summary

  • add a docs command validation script and unit coverage for implemented bundle commands, legacy resource references, and allowed docs.specfact.io routes
  • wire the validator into docs-review CI and fix the backlog docs examples it caught
  • record the docs-12 OpenSpec change, TDD evidence, and CHANGE_ORDER status

Verification

  • python3 scripts/check-docs-commands.py
  • python3 -m pytest tests/unit/docs/test_docs_review.py tests/unit/test_check_docs_commands_script.py -q
  • specfact code review run scripts/check-docs-commands.py tests/unit/test_check_docs_commands_script.py --no-tests
  • openspec validate docs-12-docs-validation-ci --strict
  • hatch run format
  • hatch run type-check
  • hatch run lint
  • hatch run yaml-lint
  • hatch run verify-modules-signature --require-signature --payload-from-filesystem --enforce-version-bump
  • hatch run contract-test
  • hatch run smart-test
  • hatch run test

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 27, 2026

Warning

Rate limit exceeded

@djm81 has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 2 minutes and 50 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 2 minutes and 50 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: c7d485f2-e3b9-44c1-8ac2-c20d24880b6c

📥 Commits

Reviewing files that changed from the base of the PR and between b54d352 and 886bace.

📒 Files selected for processing (47)
  • .github/workflows/docs-review.yml
  • docs/adapters/azuredevops.md
  • docs/adapters/backlog-adapter-patterns.md
  • docs/adapters/github.md
  • docs/getting-started/README.md
  • docs/getting-started/first-steps.md
  • docs/getting-started/installation.md
  • docs/getting-started/tutorial-backlog-refine-ai-ide.md
  • docs/getting-started/tutorial-daily-standup-sprint-review.md
  • docs/getting-started/tutorial-openspec-speckit.md
  • docs/guides/README.md
  • docs/guides/agile-scrum-workflows.md
  • docs/guides/brownfield-examples.md
  • docs/guides/command-chains.md
  • docs/guides/common-tasks.md
  • docs/guides/custom-field-mapping.md
  • docs/guides/daily-devops-routine.md
  • docs/guides/dual-stack-enrichment.md
  • docs/guides/ide-integration.md
  • docs/guides/integrations-overview.md
  • docs/guides/migration-cli-reorganization.md
  • docs/guides/migration-guide.md
  • docs/guides/module-marketplace.md
  • docs/guides/openspec-journey.md
  • docs/guides/speckit-comparison.md
  • docs/guides/template-customization.md
  • docs/guides/troubleshooting.md
  • docs/guides/use-cases.md
  • docs/guides/workflows.md
  • docs/integrations/devops-adapter-overview.md
  • docs/module-publishing-guide.md
  • docs/reference/architecture.md
  • docs/reference/command-syntax-policy.md
  • docs/reference/directory-structure.md
  • docs/reference/feature-keys.md
  • docs/reference/parameter-standard.md
  • docs/reference/specmatic.md
  • docs/reference/telemetry.md
  • docs/reference/thorough-codebase-validation.md
  • openspec/changes/docs-12-docs-validation-ci/TDD_EVIDENCE.md
  • openspec/changes/docs-12-docs-validation-ci/proposal.md
  • openspec/changes/docs-12-docs-validation-ci/specs/modules-docs-command-validation/spec.md
  • openspec/changes/docs-12-docs-validation-ci/tasks.md
  • openspec/specs/modules-docs-command-validation/spec.md
  • scripts/check-docs-commands.py
  • tests/unit/docs/test_docs_review.py
  • tests/unit/test_check_docs_commands_script.py
📝 Walkthrough

Walkthrough

This PR implements automated documentation validation CI that extracts CLI command examples from markdown files, validates them against actual Typer/Click command registrations, detects stale legacy resource references, and validates cross-site documentation links.

Changes

Cohort / File(s) Summary
CI/Workflow Configuration
.github/workflows/docs-review.yml, pyproject.toml
Added job step to run scripts/check-docs-commands.py with expanded dependency installation (typer, PyYAML, beartype, icontract, rich, pydantic, specfact-cli); workflow now triggers on script file changes. Ruff per-file ignore rule added for N999 check in the validation script.
Documentation Bundle Updates
docs/bundles/backlog/policy-engine.md, docs/bundles/backlog/refinement.md
Updated policy-engine CLI examples from specfact policy to specfact backlog policy namespace. Removed legacy template location documentation fallback from refinement docs.
Validation Script & Tests
scripts/check-docs-commands.py, tests/unit/test_check_docs_commands_script.py
Added new validation script extracting CLI examples from markdown, checking them against command registrations, detecting legacy resource paths, and validating cross-site documentation links. Tests cover command extraction, validation logic, link checking, legacy path detection, and CI workflow wiring verification.
OpenSpec Documentation
openspec/CHANGE_ORDER.md, openspec/changes/docs-12-docs-validation-ci/*, openspec/specs/modules-docs-command-validation/spec.md, openspec/specs/enterprise-config-docs/spec.md, openspec/specs/missing-command-docs/spec.md, openspec/specs/team-setup-docs/spec.md
Added new spec for modules-side docs validation, TDD evidence file documenting validation workflow, and task completion markers. Updated requirement heading levels and added formatting structure to existing specs.

Sequence Diagram

sequenceDiagram
    participant WF as CI Workflow
    participant Script as check-docs-commands.py
    participant Docs as Markdown Files
    participant Reg as Command Registry
    participant Config as docs/_config.yml
    
    WF->>Script: Execute on file changes
    Script->>Docs: Scan for markdown files
    Docs-->>Script: File content
    Script->>Script: Extract command examples
    Script->>Reg: Validate against registrations
    Reg-->>Script: Valid/Invalid status
    Script->>Docs: Check for legacy resource paths
    Docs-->>Script: Match results
    Script->>Docs: Validate links & hrefs
    Docs-->>Script: URLs found
    Script->>Config: Validate config values
    Config-->>Script: URL values
    Script->>WF: Report findings (exit 0 or 1)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~30 minutes

Possibly related issues

  • The main repository issue implements the specfact-cli/docs-12 docs-command validation CI that this cross-repo change references as its counterpart for validating module-side documentation examples against actual CLI registrations.

Possibly related PRs

Suggested labels

openspec

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ❓ Inconclusive The PR description provides a clear summary, specific verification steps, but is missing required template sections like Scope, Bundle Impact, and CI/Branch Protection checkboxes. Complete the description using the template: add Scope checkboxes, Bundle Impact section, CI/Branch Protection checklist, and Docs/Pages section to align with repository standards.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title 'docs: add docs validation ci checks' directly and clearly summarizes the main change: adding documentation validation to CI.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/docs-12-docs-validation-ci

Comment @coderabbitai help to get the list of available commands and usage tips.

@chatgpt-codex-connector
Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@djm81 djm81 self-assigned this Mar 27, 2026
@djm81 djm81 added documentation Improvements or additions to documentation enhancement New feature or request labels Mar 27, 2026
@djm81 djm81 moved this from Todo to In Progress in SpecFact CLI Mar 27, 2026
@djm81 djm81 linked an issue Mar 27, 2026 that may be closed by this pull request
@djm81
Copy link
Copy Markdown
Contributor Author

djm81 commented Mar 27, 2026

@coderabbitai review the updated PR

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 27, 2026

@djm81 Sure! I'll kick off a full review of the updated PR now.


🧠 Learnings used
Learnt from: CR
Repo: nold-ai/specfact-cli-modules PR: 0
File: CLAUDE.md:0-0
Timestamp: 2026-03-25T21:31:11.712Z
Learning: Follow strict TDD order: spec delta -> failing tests -> implementation -> passing tests -> quality gates. Record TDD evidence in `openspec/changes/<change-id>/TDD_EVIDENCE.md`

Learnt from: CR
Repo: nold-ai/specfact-cli-modules PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-03-26T23:11:11.388Z
Learning: Before changing code, verify an active OpenSpec change explicitly covers the requested scope; follow strict TDD order: spec delta → failing tests → implementation → passing tests → quality gates

Learnt from: CR
Repo: nold-ai/specfact-cli-modules PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-03-26T23:11:11.388Z
Learning: Applies to openspec/changes/*/TDD_EVIDENCE.md : Record failing/passing test evidence in openspec/changes/<change-id>/TDD_EVIDENCE.md
✅ Actions performed

Full review triggered.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In @.github/workflows/docs-review.yml:
- Line 51: The pip install line in the GitHub Actions workflow misses an
explicit click dependency even though scripts/check-docs-commands.py imports
click; update the pip install command (the line installing pytest typer PyYAML
beartype icontract rich pydantic specfact-cli) to also include click so the
workflow explicitly installs click alongside the other packages, ensuring
scripts/check-docs-commands.py has its runtime dependency satisfied.

In `@scripts/check-docs-commands.py`:
- Around line 104-117: The fence detection in _iter_bash_examples is too strict:
change the checks that compare stripped == "```bash" and stripped == "```" to
more permissive startswith checks (e.g., use stripped.startswith("```bash") and
stripped.startswith("```")) so that trailing spaces or attributes on the fence
line are accepted; update the in_bash_block toggles around those checks
(in_bash_block variable and the block that appends CommandExample) so behavior
remains identical except for relaxed fence matching.
- Around line 195-211: Multiple functions (_validate_command_examples,
_validate_legacy_resource_paths, _validate_core_docs_links) reread the same
files; refactor to read each file once and reuse its contents by introducing a
cached mapping (e.g., dict[Path, str]) or by changing call sites to pass file
text to these validators; update _validate_legacy_resource_paths to accept
either a mapping or the file text rather than calling Path.read_text itself
(refer to function name _validate_legacy_resource_paths and the other
validators) and use that shared cache when iterating snippets so each file is
read only once.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: fe50fdbe-5768-40f7-9f98-82bb3228b07a

📥 Commits

Reviewing files that changed from the base of the PR and between 773e512 and b54d352.

📒 Files selected for processing (14)
  • .github/workflows/docs-review.yml
  • docs/bundles/backlog/policy-engine.md
  • docs/bundles/backlog/refinement.md
  • openspec/CHANGE_ORDER.md
  • openspec/changes/docs-12-docs-validation-ci/TDD_EVIDENCE.md
  • openspec/changes/docs-12-docs-validation-ci/specs/modules-docs-command-validation/spec.md
  • openspec/changes/docs-12-docs-validation-ci/tasks.md
  • openspec/specs/enterprise-config-docs/spec.md
  • openspec/specs/missing-command-docs/spec.md
  • openspec/specs/modules-docs-command-validation/spec.md
  • openspec/specs/team-setup-docs/spec.md
  • pyproject.toml
  • scripts/check-docs-commands.py
  • tests/unit/test_check_docs_commands_script.py
💤 Files with no reviewable changes (1)
  • docs/bundles/backlog/refinement.md

@djm81 djm81 merged commit 45d0585 into dev Mar 27, 2026
10 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in SpecFact CLI Mar 27, 2026
@coderabbitai coderabbitai bot mentioned this pull request Mar 28, 2026
23 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

docs-12: Add CI Validation For Docs Command Examples (Modules Side)

1 participant