-
Notifications
You must be signed in to change notification settings - Fork 2
Infra/mlo 3 hermetic workspaces docker #635
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| .git | ||
| .github | ||
| .venv | ||
| venv | ||
| __pycache__ | ||
| .pytest_cache | ||
| .coverage | ||
| .coverage.* | ||
| htmlcov | ||
| build | ||
| neuraldbg.egg-info | ||
| artifacts | ||
| data | ||
| models | ||
| outputs | ||
| *.log | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| FROM python:3.12-slim | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: This workspace is not actually hermetic because the base image and critical Python tooling are left unpinned. A rebuild later can resolve different images or wheels and break reproducibility without any repo change. Prompt for AI agents |
||
|
|
||
| ENV PIP_NO_CACHE_DIR=1 \ | ||
| PYTHONDONTWRITEBYTECODE=1 \ | ||
| PYTHONUNBUFFERED=1 | ||
|
|
||
| WORKDIR /workspace | ||
|
|
||
| COPY requirements.txt requirements-dev.txt ./ | ||
|
|
||
| RUN apt-get update && \ | ||
|
Check warning on line 11 in Dockerfile
|
||
| apt-get install -y --no-install-recommends build-essential git && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
|
|
||
| RUN python -m pip install --upgrade pip && \ | ||
| python -m pip install --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple torch && \ | ||
| python -m pip install -r requirements.txt && \ | ||
| python -m pip install -r requirements-dev.txt | ||
|
|
||
| COPY . . | ||
|
|
||
| CMD ["bash"] | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| SHELL := /bin/bash | ||
|
|
||
| PYTHON := $(shell if [ -x .venv/bin/python ]; then echo .venv/bin/python; elif command -v python3 >/dev/null 2>&1; then echo python3; else echo python; fi) | ||
| COMPOSE := docker-compose | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Hardcoding Prompt for AI agents |
||
|
|
||
| .PHONY: help install install-dev up down build rebuild shell test test-docker coverage bandit safety security precommit clean | ||
|
|
||
| help: | ||
| @echo "NeuralDBG Make targets:" | ||
| @echo " make install - install runtime dependencies" | ||
| @echo " make install-dev - install runtime + dev dependencies" | ||
| @echo " make build - build Docker dev image" | ||
| @echo " make up - start Docker dev service" | ||
| @echo " make down - stop Docker dev service" | ||
| @echo " make shell - open shell in running dev container" | ||
| @echo " make test - run pytest locally" | ||
| @echo " make test-docker - run pytest in Docker container" | ||
| @echo " make coverage - run coverage with gate >= 60%" | ||
| @echo " make bandit - run Bandit security scan" | ||
| @echo " make safety - run Safety dependency scan" | ||
| @echo " make security - run Bandit + Safety" | ||
| @echo " make precommit - run pre-commit hooks on all files" | ||
| @echo " make clean - remove local QA artifacts" | ||
|
|
||
| install: | ||
| $(PYTHON) -m pip install --upgrade pip | ||
| $(PYTHON) -m pip install --index-url https://download.pytorch.org/whl/cpu --extra-index-url https://pypi.org/simple torch | ||
| $(PYTHON) -m pip install -r requirements.txt | ||
|
|
||
| install-dev: install | ||
| $(PYTHON) -m pip install -r requirements-dev.txt | ||
|
|
||
| build: | ||
| $(COMPOSE) build neuraldbg-dev | ||
|
|
||
| rebuild: | ||
| $(COMPOSE) build --no-cache neuraldbg-dev | ||
|
|
||
| up: | ||
| $(COMPOSE) up -d | ||
|
|
||
| down: | ||
| $(COMPOSE) down | ||
|
|
||
| shell: | ||
| $(COMPOSE) exec neuraldbg-dev bash | ||
|
|
||
| test: | ||
| $(PYTHON) -m pytest | ||
|
|
||
| test-docker: | ||
| $(COMPOSE) run --rm neuraldbg-dev bash -lc "pytest" | ||
|
|
||
| coverage: | ||
| $(PYTHON) -m coverage run --source=neuraldbg -m pytest | ||
| $(PYTHON) -m coverage report --show-missing --fail-under=60 | ||
|
|
||
| bandit: | ||
| $(PYTHON) -m bandit -r . -ll -x tests,*/tests/*,.venv,venv,__pycache__ | ||
|
|
||
| safety: | ||
| $(PYTHON) -m safety check --full-report | ||
|
|
||
| security: bandit safety | ||
|
|
||
| precommit: | ||
| $(PYTHON) -m pre_commit run --all-files | ||
|
|
||
| clean: | ||
| rm -rf .pytest_cache htmlcov .mypy_cache | ||
| rm -f .coverage .coverage.* | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
|
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| version: "3.9" | ||
|
|
||
| services: | ||
| neuraldbg-dev: | ||
| build: | ||
| context: . | ||
| dockerfile: Dockerfile | ||
| working_dir: /workspace | ||
| command: bash -lc "sleep infinity" | ||
| tty: true | ||
| stdin_open: true | ||
| volumes: | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P2: Add a host-user mapping for the bind-mounted workspace; otherwise container writes will leave root-owned files in the repo. Prompt for AI agents |
||
| - ./:/workspace | ||
| - ./data:/data | ||
| - ./models:/models | ||
| - ./outputs:/outputs | ||
|
Comment on lines
+8
to
+16
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 1. Root-owned bind mount The dev container runs as root while bind-mounting ./ into /workspace, so running pytest/coverage inside Docker will create root-owned files on the host (e.g., .pytest_cache, htmlcov, .coverage) and can break local cleanup/editing without sudo. This is triggered directly by make test-docker / docker-compose run workflows. Agent Prompt
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
|
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
|
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| pytest>=8.0.0,<9.0.0 | ||
| coverage>=7.0.0,<8.0.0 | ||
| bandit>=1.7.0,<2.0.0 | ||
| safety>=3.0.0,<4.0.0 | ||
| pre-commit>=3.0.0,<4.0.0 | ||
|
Comment on lines
+1
to
+5
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Verify whether dependency resolution is deterministic or range-based.
echo "== Dependency manifests =="
fd 'requirements.*\.txt|constraints.*\.txt|poetry\.lock|Pipfile\.lock|uv\.lock'
echo
echo "== Version specifiers inside requirements files =="
rg -n '^[a-zA-Z0-9_.-]+(>=|<=|~=|>|<)' --glob 'requirements*.txt' --glob 'constraints*.txt'
echo
echo "== Install commands (Dockerfile / Makefile / scripts) =="
rg -n 'pip install .*requirements(-dev)?\.txt|--require-hashes|-c constraints' Dockerfile MakefileRepository: LambdaSection/NeuralDBG Length of output: 749 Dependency resolution is not deterministic; add a lock file to guarantee reproducible builds. The project lacks a lock mechanism (no poetry.lock, requirements.lock, or constraints file). While 🤖 Prompt for AI Agents |
||
| Original file line number | Diff line number | Diff line change | ||||||
|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1 @@ | ||||||||
| numpy>=1.26.0 | ||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. P1: Prompt for AI agents
Suggested change
|
||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add
.envand secrets patterns to prevent accidental inclusion in Docker images.The
.dockerignorefile should exclude.env,.env.*, andsecrets/to prevent sensitive data from being copied into Docker images via theCOPY . .instruction in the Dockerfile.Proposed fix
Based on learnings: "Certain files MUST be in .gitignore and NEVER committed publicly: ... .env, and API keys/tokens"
📝 Committable suggestion
🤖 Prompt for AI Agents