Skip to content

Develop #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bartzbeielstein merged 7 commits into from
Apr 3, 2026
Merged

Develop #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ff400e0
fix: labels
bartzbeielstein Apr 2, 2026
497ae0d
fix: ci
bartzbeielstein Apr 2, 2026
39ebe63
fix: renaming fit()
bartzbeielstein Apr 2, 2026
e976ced
fix: quartodoc ru
bartzbeielstein Apr 2, 2026
0c23ab5
Update spotoptim_class.qmd
bartzbeielstein Apr 2, 2026
4c4837b
fix: quartodoc 2nd
bartzbeielstein Apr 3, 2026
b2ccdb7
Update optimize_parallel.qmd
bartzbeielstein Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
284 changes: 83 additions & 201 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,43 +1,3 @@
# SPDX-FileCopyrightText: 2026 bartzbeielstein
#
# SPDX-License-Identifier: AGPL-3.0-or-later
#
# =============================================================================
# .github/workflows/ci.yml — Continuous Integration (Optimised v2)
# =============================================================================
#
# Performance optimisations (cumulative with v1):
# 1. Concurrency groups → auto-cancel superseded runs on same branch
# 2. Path-based filtering → skip tests entirely for docs-only changes
# 3. 8-shard matrix → 127 test files ÷ 8 ≈ 16 files per shard
# 4. pytest-xdist (-n auto) → each shard uses all available cores (~4×)
# 5. CPU-only PyTorch → ~800 MB smaller install, 2-3 min saved
# 6. Full .venv caching → near-instant installs on cache hit
# 7. Ruff-only lint → single tool replaces black + isort + ruff
# 8. Job-level timeouts → prevent stuck tests from burning hours
# 9. Pytest --timeout → per-test 5-minute guard via pytest-timeout
# 10. Consolidated quality job → lint + security in one job, one venv
#
# Jobs:
# changes — detect which paths changed (gate for test/quality)
# reuse — REUSE licence compliance (always)
# test — pytest + coverage upload, 8 shards (skip for docs-only)
# coverage — merge shard reports & upload to Codecov
# quality — ruff + bandit (skip for docs-only, non-blocking)
# =============================================================================
#
# Jobs:
# reuse — REUSE licence compliance (PR + push)
# test — pytest + coverage upload (PR + push, Python 3.13)
# lint — black / isort / ruff formatting checks (non-blocking)
# security — safety + bandit vulnerability scans (non-blocking)
#
# Note: Linear-history enforcement (no merge commits) is handled by GitHub's
# built-in "Require linear history" branch-protection rule, which is faster,
# more reliable, and blocks the merge button before CI even starts.
# The previous `no-merge-commits` CI job has been removed.
# =============================================================================

name: CI Tests

on:
Expand All @@ -50,224 +10,146 @@ on:
- main
- develop

# Cancel in-flight runs on the same branch/PR — the single biggest time saver
# when multiple pushes happen in quick succession.
# Principle of Least Privilege: Base permissions are read-only
permissions: read-all

# Cancel stale runs when a new push arrives on the same branch or PR.
concurrency:
group: ci-${{ github.ref }}
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

permissions:
contents: read

env:
PYTHON_VERSION: "3.13"
# PyTorch CPU-only index is configured in pyproject.toml [tool.uv.sources]

jobs:

# ---------------------------------------------------------------------------
# Detect changed paths to gate downstream jobs
# ---------------------------------------------------------------------------
changes:
name: Detect Changes
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: read
outputs:
code: ${{ steps.filter.outputs.code }}
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- name: Check changed paths
uses: dorny/paths-filter@9d7afb8d214ad99e78fbd4247752c4caed2b6e4c # v4.0.0
id: filter
with:
filters: |
code:
- 'src/**'
- 'tests/**'
- 'pyproject.toml'
- 'uv.lock'
- '.github/workflows/ci.yml'

# ---------------------------------------------------------------------------
# REUSE licence compliance (always runs — fast, no deps)
# ---------------------------------------------------------------------------
reuse:
name: REUSE Compliance
runs-on: ubuntu-latest
permissions:
contents: read

steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- name: Set up uv
uses: astral-sh/setup-uv@e06108dd0aef18192324c70427afc47652e63a82 # v7.5.0
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5.2.2

- name: Run reuse lint
run: uv tool run reuse lint
run: |
uv tool run reuse lint

# ---------------------------------------------------------------------------
# Test matrix — 8 shards, each running pytest-xdist on all available cores
# ---------------------------------------------------------------------------
test:
name: "Test (shard ${{ matrix.shard }}/8)"
name: Test on Python ${{ matrix.python-version }}
runs-on: ubuntu-latest
needs: changes
if: needs.changes.outputs.code == 'true'
timeout-minutes: 90
permissions:
contents: read
id-token: write

strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8]
python-version: ["3.13"]

steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
python-version: ${{ env.PYTHON_VERSION }}
fetch-depth: 0

- name: Set up uv
uses: astral-sh/setup-uv@e06108dd0aef18192324c70427afc47652e63a82 # v7.5.0
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
python-version: ${{ matrix.python-version }}

# Cache the full .venv so subsequent runs skip dependency resolution
- name: Cache virtual environment
uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
with:
path: .venv
key: venv-test-${{ runner.os }}-py${{ env.PYTHON_VERSION }}-${{ hashFiles('uv.lock') }}
restore-keys: |
venv-test-${{ runner.os }}-py${{ env.PYTHON_VERSION }}-
- name: Set up uv
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5.2.2

- name: Install dependencies
run: uv sync --all-extras --group dev

# Simple modulo-based file assignment — no pytest --collect-only overhead
# (saves 30–60 s per shard × 8 shards ≈ 4–8 min of wasted collection time).
- name: Select tests for shard ${{ matrix.shard }}
id: shard
run: |
shard=${{ matrix.shard }}
num_shards=8

mapfile -t all_files < <(ls tests/test_*.py 2>/dev/null | sort)
shard_files=()
for i in "${!all_files[@]}"; do
if (( (i % num_shards) + 1 == shard )); then
shard_files+=("${all_files[$i]}")
fi
done

if [ ${#shard_files[@]} -eq 0 ]; then
echo "files=" >> "$GITHUB_OUTPUT"
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "::notice::Shard ${shard}: no test files assigned"
else
echo "files=${shard_files[*]}" >> "$GITHUB_OUTPUT"
echo "skip=false" >> "$GITHUB_OUTPUT"
echo "::notice::Shard ${shard}: ${#shard_files[@]} test files"
fi
uv pip install --system -e ".[dev]"

- name: Run pytest (shard ${{ matrix.shard }})
if: steps.shard.outputs.skip != 'true'
- name: Run pytest
run: |
uv run pytest ${{ steps.shard.outputs.files }} \
-n auto \
--dist loadfile \
--tb=short \
--timeout=900 \
--cov=src/spotoptim \
--cov-report=xml:coverage-shard${{ matrix.shard }}.xml \
--cov-report=term
uv run pytest tests/ -v -n auto --tb=short --cov=src/spotoptim --cov-branch --cov-report=xml --cov-report=term

- name: Upload shard coverage
if: steps.shard.outputs.skip != 'true'
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
- name: Upload results to Codecov
uses: codecov/codecov-action@1af58845a975a7985b0beb0cbe6fbbb71a41dbad # v5.5.3
with:
name: coverage-shard-${{ matrix.shard }}
path: coverage-shard${{ matrix.shard }}.xml
retention-days: 1
token: ${{ secrets.CODECOV_TOKEN }}
slug: sequential-parameter-optimization/spotoptim
files: ./coverage.xml
fail_ci_if_error: false

# ---------------------------------------------------------------------------
# Merge coverage from all shards and upload to Codecov
# ---------------------------------------------------------------------------
coverage:
name: Upload Coverage
lint:
name: Code Quality
runs-on: ubuntu-latest
needs: test
if: always() && needs.test.result == 'success'
permissions:
contents: read

steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- name: Download all coverage artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
pattern: coverage-shard-*
merge-multiple: true
python-version: "3.13"

- name: Upload coverage to Codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5.5.2
with:
files: coverage-shard1.xml,coverage-shard2.xml,coverage-shard3.xml,coverage-shard4.xml,coverage-shard5.xml,coverage-shard6.xml,coverage-shard7.xml,coverage-shard8.xml
fail_ci_if_error: false
- name: Set up uv
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5.2.2

- name: Install dependencies
run: |
uv pip install --system -e ".[dev]"

- name: Check formatting with black
run: |
black --check src/ tests/
continue-on-error: true

- name: Check imports with isort
run: |
isort --check-only src/ tests/
continue-on-error: true

- name: Lint with ruff
run: |
ruff check src/ tests/
continue-on-error: true

# ---------------------------------------------------------------------------
# Code quality + security (consolidated, non-blocking)
# ---------------------------------------------------------------------------
quality:
name: Code Quality & Security
security:
name: Security Scan
runs-on: ubuntu-latest
needs: changes
if: needs.changes.outputs.code == 'true'
timeout-minutes: 10
permissions:
contents: read
security-events: write

steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Checkout code
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2

- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: ${{ env.PYTHON_VERSION }}
python-version: "3.13"

- name: Set up uv
uses: astral-sh/setup-uv@e06108dd0aef18192324c70427afc47652e63a82 # v7.5.0
with:
enable-cache: true
cache-dependency-glob: "uv.lock"
uses: astral-sh/setup-uv@e58605a9b6da7c637471fab8847a5e5a6b8df081 # v5.2.2

# Lint/security only needs dev tools, not the full project with torch
- name: Install lint tools only
run: uv sync --group dev --no-install-project
- name: Install dependencies
run: |
uv pip install --system -e ".[dev]"

- name: Check formatting (ruff format)
run: uv run ruff format --check src/ tests/
- name: Check dependencies with safety
run: |
safety check --json
continue-on-error: true

- name: Lint (ruff check)
run: uv run ruff check src/ tests/
- name: Security scan with bandit
run: |
bandit -r src/spotoptim/ -f sarif -o bandit-report.sarif
continue-on-error: true

- name: Static security analysis (bandit)
# --exit-zero: exit 0 even when issues are found so the step stays green.
# Findings are still printed to the log; this job is advisory only.
run: uv run bandit -r src/spotoptim/ --exit-zero
- name: Upload bandit report to security tab
if: always()
uses: github/codeql-action/upload-sarif@c6f931105cb2c34c8f901cc885ba1e2e259cf745 # v4.34.0
with:
sarif_file: bandit-report.sarif
category: bandit
continue-on-error: true
Loading
Loading