[Testing] Documentation Code Validation

[itomek-amd]

Overview

Automated validation of all Python code snippets in documentation to ensure examples work as documented.

Scope

• Create util/doc_snippet_runner.py to parse and validate MDX code snippets
• Create tests_v2/docs/test_doc_snippets.py with auto-generated tests
• Add .github/workflows/test_doc_snippets.yml CI workflow

Implementation

1. Create Doc Snippet Parser

File: util/doc_snippet_runner.py

Features:

• Parse all MDX files in docs/ for Python code blocks
• Classify snippets (import-only, syntax-only, executable, interactive)
• Generate pytest test file from runnable snippets
• Support skip annotations in MDX (<!-- skip-test -->)

2. Create Generated Tests

File: tests_v2/docs/test_doc_snippets.py

Auto-generated parametrized tests for all code snippets from documentation.

3. Create CI Workflow

File: .github/workflows/test_doc_snippets.yml

name: Documentation Code Validation
on:
  push:
    paths: ['docs/**/*.mdx']
  pull_request:
    paths: ['docs/**/*.mdx']

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: '3.12'
      - run: pip install -e ".[dev]"
      - run: python util/doc_snippet_runner.py --generate
      - run: pytest tests_v2/docs/test_doc_snippets.py -v

Documentation to Validate

95 MDX files with 826+ Python code blocks across:

• Quickstart guides
• SDK documentation
• Agent guides
• API references

Acceptance Criteria

• util/doc_snippet_runner.py created and functional
• tests_v2/docs/test_doc_snippets.py generated successfully
• .github/workflows/test_doc_snippets.yml created
• All runnable documentation snippets validated
• CI workflow runs on MDX changes

Parent Issue

Part of #209 - Test Suite Overhaul Epic

Dependencies

Requires #239 (tests_v2 foundation) to be completed first.