Feat/implement multi database support

.github/workflows/python-test.yml

@@ -34,17 +34,10 @@ jobs:
           python -m pip install -r requirements.txt
           python -m pip install -e .[dev]
 
-      - name: Lint with flake8
+      - name: Lint and check with ruff
         run: |
-          flake8 . --count --show-source --statistics
-
-      - name: Check typing with mypy
-        run: |
-          mypy --install-types --non-interactive .
-
-      - name: Check syntax with pyupgrade
-        run: |
-          find . -type f -regex '.*\.py$' -exec pyupgrade --py311-plus {} \;
+          ruff check . --output-format=github
+          ruff format --check .
 
       - name: Set up Helm
         uses: azure/setup-helm@v1
@@ -76,4 +69,4 @@ jobs:
         with:
           github_token: ${{ secrets.github_token }}
           # Change reviewdog reporter if you need [github-pr-check, github-check].
-          reporter: github-pr-check
+          reporter: github-pr-check

.gitignore

@@ -68,3 +68,7 @@ venv-python3.7
 venv-python3.8
 venv-python3.10
 .python-version
+
+# Helm Chart 
+*.lock
+example/helm-chart/my-example-chart/charts/

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## v6.0.0 (2025-10-07)
+
+### Feat
+- add support for multi-database migration
+
+
 ## v5.0.0 (2025-05-05)
 ### BREAKING CHANGE
 - Change minimum supported version of python: 3.11 (drop 3.7, 3.8, 3.9, 3.10)

VERSION

@@ -1 +1 @@
-5.0.0
+6.0.0

docs/multi-database.md

@@ -0,0 +1,166 @@
+# Multi-Database Migration Support
+
+Chartreuse now supports managing migrations for multiple databases simultaneously. This feature allows you to define multiple database configurations and run migrations across all of them.
+
+## Configuration
+
+### Single Database (Legacy)
+The original single-database configuration using environment variables is still supported for backward compatibility.
+
+### Multiple Databases (New)
+To use multiple databases, create a YAML configuration file and set the `CHARTREUSE_MULTI_CONFIG_PATH` environment variable to point to it.
+
+#### Configuration File Format
+
+```yaml
+databases:
+  # Main application database
+  main:
+    alembic_directory_path: /app/alembic/main
+    alembic_config_file_path: alembic.ini
+    dialect: postgresql
+    user: app_user
+    password: app_password
+    host: postgres-main
+    port: 5432
+    database: app_main
+    allow_migration_for_empty_database: true
+    additional_parameters: ""
+
+  # Analytics database
+  analytics:
+    alembic_directory_path: /app/alembic/analytics
+    alembic_config_file_path: alembic.ini
+    dialect: postgresql
+    user: analytics_user
+    password: analytics_password
+    host: postgres-analytics
+    port: 5432
+    database: analytics
+    allow_migration_for_empty_database: false
+    additional_parameters: ""
+```
+
+#### Configuration Fields
+
+**Required fields:**
+- Database name (used as key in the YAML)
+- `alembic_directory_path`: Path to the Alembic directory for this database
+- `alembic_config_file_path`: Alembic configuration file name
+
+**Database connection components (all required):**
+- `dialect`: Database dialect (e.g., postgresql, mysql, sqlite)
+- `user`: Database username
+- `password`: Database password
+- `host`: Database host
+- `port`: Database port
+- `database`: Database name
+
+**Optional fields:**
+- `allow_migration_for_empty_database`: Whether to allow migrations on empty databases (default: false)
+- `additional_parameters`: Additional parameters to pass to Alembic commands (default: "")
+
+## Environment Variables
+
+For multi-database mode:
+- `CHARTREUSE_MULTI_CONFIG_PATH`: Path to the YAML configuration file
+- `CHARTREUSE_ENABLE_STOP_PODS`: Whether to stop pods during migration (optional, default: true)
+- `CHARTREUSE_RELEASE_NAME`: Kubernetes release name
+- `CHARTREUSE_UPGRADE_BEFORE_DEPLOYMENT`: Whether to upgrade before deployment (optional, default: false)
+- `HELM_IS_INSTALL`: Whether this is a Helm install operation (optional, default: false)
+
+## Usage
+
+### Setting up the environment
+```bash
+export CHARTREUSE_MULTI_CONFIG_PATH="/path/to/multi-database-config.yaml"
+export CHARTREUSE_RELEASE_NAME="my-app"
+export CHARTREUSE_ENABLE_STOP_PODS="true"
+```
+
+### Directory Structure Example
+```
+/app/
+├── alembic/
+│   ├── main/
+│   │   ├── alembic.ini
+│   │   ├── env.py
+│   │   └── versions/
+│   └── analytics/
+│       ├── alembic.ini
+│       ├── env.py
+│       └── versions/
+└── multi-database-config.yaml
+```
+
+### Configuration Validation
+You can validate your configuration file before deployment:
+
+```bash
+python3 scripts/validate_config.py /path/to/multi-database-config.yaml
+```
+
+## Migration Behavior
+
+When using multi-database configuration:
+
+1. **Initialization**: All databases are initialized with their respective Alembic configurations
+2. **Migration Check**: Each database is checked individually for pending migrations
+3. **Migration Execution**: Only databases that need migration will be upgraded
+4. **Error Handling**: If any database migration fails, the entire process fails
+5. **Logging**: Detailed logs show which databases are being processed
+
+## Backward Compatibility
+
+The original single-database configuration using environment variables continues to work unchanged. The multi-database feature is only activated when `CHARTREUSE_MULTI_CONFIG_PATH` is set.
+
+## Example Integration
+
+In your Helm chart, you can use a ConfigMap to store the multi-database configuration:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: chartreuse-config
+data:
+  multi-database-config.yaml: |
+    databases:
+      main:
+        alembic_directory_path: /app/alembic/main
+        alembic_config_file_path: alembic.ini
+        dialect: postgresql
+        user: {{ .Values.database.main.user }}
+        password: {{ .Values.database.main.password }}
+        host: {{ .Values.database.main.host }}
+        port: {{ .Values.database.main.port }}
+        database: {{ .Values.database.main.name }}
+        allow_migration_for_empty_database: true
+        additional_parameters: ""
+      analytics:
+        alembic_directory_path: /app/alembic/analytics
+        alembic_config_file_path: alembic.ini
+        dialect: postgresql
+        user: {{ .Values.database.analytics.user }}
+        password: {{ .Values.database.analytics.password }}
+        host: {{ .Values.database.analytics.host }}
+        port: {{ .Values.database.analytics.port }}
+        database: {{ .Values.database.analytics.name }}
+        allow_migration_for_empty_database: false
+        additional_parameters: ""
+```
+
+Then mount it in your migration job:
+
+```yaml
+env:
+- name: CHARTREUSE_MULTI_CONFIG_PATH
+  value: "/config/multi-database-config.yaml"
+volumeMounts:
+- name: config
+  mountPath: /config
+volumes:
+- name: config
+  configMap:
+    name: chartreuse-config
+```

example/Dockerfile-dev

@@ -1,12 +1,16 @@
 # Only used for e2e tests. Do not use in production.
-
+# For Wiremind use only since it uses our private package wiremind-python
 FROM python:3.12
 
 WORKDIR /app
 
+# Install uv
+RUN pip install uv
+
 COPY requirements.txt requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt
+RUN uv pip install --system --no-cache -r requirements.txt
 
 COPY . .
-RUN pip install -e .
-COPY example/alembic alembic
+RUN uv pip install --system -e .
+RUN uv pip install --system wiremind-python
+COPY alembic alembic

example/Dockerfile-dev-auth

@@ -0,0 +1,28 @@
+# Only used for e2e tests. Do not use in production.
+# For Wiremind use only since it uses our private package wiremind-python
+
+FROM python:3.12
+
+WORKDIR /app
+
+# Install uv
+RUN pip install uv
+
+# Install dependencies if requirements.txt exists
+COPY pyproject.toml .
+COPY VERSION .
+COPY README.md .
+COPY requirements.txt* ./
+COPY src/ src/
+COPY example/alembic/ alembic/
+
+RUN if [ -f requirements.txt ]; then uv pip install --system --no-cache -r requirements.txt; fi
+
+# Install the package itself
+RUN uv pip install --system -e .
+
+# Install wiremind-python with authentication using build secrets
+# The secret will be mounted at build time but not stored in the image
+RUN --mount=type=secret,id=uv_config,target=/root/.config/uv/uv.toml \
+    mkdir -p /root/.config/uv && \
+    uv pip install --system wiremind-python cyrillic clickhouse-sqlalchemy

example/alembic/alembic.ini

@@ -1,55 +1,17 @@
-# A generic, single database configuration.
-
-[alembic]
-# path to migration scripts
-script_location = .
-
+[postgresql]
+script_location = postgresl
+sqlalchemy.url = postgresql://wiremind_owner@localhost:5432/wiremind
+prepend_sys_path = ..
+file_template = %%(year)d%%(month).2d%%(day).2d-%%(slug)s
+
+[clickhouse]
+script_location = clickhouse
+sqlalchemy.url = clickhouse://default@localhost:8123/wiremind
+prepend_sys_path = ..
 # template used to generate migration files
-# file_template = %%(rev)s_%%(slug)s
-
-# timezone to use when rendering the date
-# within the migration file as well as the filename.
-# string value is passed to dateutil.tz.gettz()
-# leave blank for localtime
-# timezone =
-
-# max length of characters to apply to the
-# "slug" field
-# truncate_slug_length = 40
-
-# set to 'true' to run the environment during
-# the 'revision' command, regardless of autogenerate
-# revision_environment = false
-
-# set to 'true' to allow .pyc and .pyo files without
-# a source .py file to be detected as revisions in the
-# versions/ directory
-# sourceless = false
-
-# version location specification; this defaults
-# to sample_alembic/versions.  When using multiple version
-# directories, initial revisions must be specified with --version-path
-# version_locations = %(here)s/bar %(here)s/bat sample_alembic/versions
-
-# the output encoding used when revision files
-# are written from script.py.mako
-# output_encoding = utf-8
-
-sqlalchemy.url=postgresql://foo:foo@localhost/foo?sslmode=prefer
-
-
-[post_write_hooks]
-# post_write_hooks defines scripts or Python functions that are run
-# on newly generated revision scripts.  See the documentation for further
-# detail and examples
-
-# format using "black" - use the console_scripts runner, against the "black" entrypoint
-# hooks=black
-# black.type=console_scripts
-# black.entrypoint=black
-# black.options=-l 79
+file_template = %%(year)d%%(month).2d%%(day).2d-%%(slug)s
 
-# Logging configuration
+# Logging configuration - Required for fileConfig() in env.py
 [loggers]
 keys = root,sqlalchemy,alembic
 
@@ -82,4 +44,4 @@ formatter = generic
 
 [formatter_generic]
 format = %(levelname)-5.5s [%(name)s] %(message)s
-datefmt = %H:%M:%S
+datefmt = %H:%M:%S