From 3f0f89032d8dca396f2379d22dd712fdf1868cf9 Mon Sep 17 00:00:00 2001 From: Renaud Cepre <32103211+renaudcepre@users.noreply.github.com> Date: Sun, 22 Mar 2026 10:15:38 +0100 Subject: [PATCH] docs: remove fixture scoping design document and simplify README - Deleted `fixture-scoping-design.md`, as it is no longer relevant. - Updated README: - Consolidated sections on async, parallelism, and smart tagging for improved readability. - Removed redundant examples, outdated parameterization details, and fixture scoping explanations. - Highlighted key features and added a link to full documentation. --- README.md | 427 ++------------------ docs/architecture/fixture-scoping-design.md | 273 ------------- 2 files changed, 25 insertions(+), 675 deletions(-) delete mode 100644 docs/architecture/fixture-scoping-design.md diff --git a/README.md b/README.md index f99def8..32af39d 100644 --- a/README.md +++ b/README.md @@ -12,18 +12,6 @@ Modern, async-first testing framework for Python 3.10+ ## Why ProTest? -ProTest isn't just another test runner. It's built for **modern Python**: typed, async, -and explicit. - -### Native I/O Concurrency - -Tests run as coroutines on a single event loop. Parallel I/O tests without spawning -separate processes. - -```bash -protest run tests:session -n 10 -``` - ### Explicit Injection (IDE-Ready) **Ctrl+Click works.** Your IDE knows every type. No guessing where fixtures come from. @@ -32,10 +20,17 @@ protest run tests:session -n 10 def test_user(db: Annotated[Database, Use(database)]): ... ``` +### Native Async & Parallelism + +Tests run as coroutines on a single event loop. No plugin needed. + +```bash +protest run tests:session -n 10 +``` + ### Smart Tagging (Tag Propagation) -Tag a fixture once, and **every test using it inherits the tag automatically**—even -through deep dependency chains. +Tag a fixture once, every test using it inherits the tag automatically. ```python @fixture(tags=["database"]) @@ -43,13 +38,6 @@ def db(): ... session.bind(db) - -@fixture() -def user_repo(db: Annotated[DB, Use(db)]): ... # Inherits "database" tag - -session.bind(user_repo) - - @session.test() def test_users(repo: Annotated[Repo, Use(user_repo)]): ... # Also tagged "database" ``` @@ -58,87 +46,22 @@ def test_users(repo: Annotated[Repo, Use(user_repo)]): ... # Also tagged "datab protest run tests:session --no-tag database # Skips ALL tests touching DB ``` -*No manual tagging. No forgotten markers.* - ### Infra vs Code Errors (Error ≠ Fail) -Know instantly if your **code** failed or your **infrastructure** crashed. - ``` ✗ test_create_user: AssertionError # Your bug - TEST FAILED ⚠ test_with_db: [FIXTURE] ConnectionError # Infra issue - SETUP ERROR ``` -### Typed Parameterization (`From` + `ForEach`) - -No more string-matching that breaks when you rename arguments. +### Typed Parameterization ```python -# pytest: Renaming "code" causes a runtime error -@pytest.mark.parametrize("code", [200, 201]) -def test_status(code): ... - - -# ProTest: Rename-safe, IDE-aware, type-checked CODES = ForEach([200, 201]) - @session.test() def test_status(code: Annotated[int, From(CODES)]): ... ``` -### Tree-Based Scoping (Smart Teardown) - -Resources are cleaned up **as soon as they're no longer needed**, not at the end. Nested -suites with predictive cleanup. - -```python -session -├── api_suite # api_client cleaned when api_suite ends -│ └── admin_suite # admin_token cleaned when admin_suite ends -└── unit_suite # No DB loaded if no test needs it (lazy) -``` - -### Managed Factories (`@factory`) - -Create test data with automatic caching and cleanup. No manual teardown. - -```python -@factory() -def user(name: str): - user = User.create(name=name) - yield user - user.delete() # Auto-cleanup for EACH created instance - -session.bind(user) - - -@session.test() -async def test_users(user_factory: Annotated[FixtureFactory[User], Use(user)]): - alice = await user_factory(name="alice") # Cached - bob = await user_factory(name="bob") # New instance - # Both cleaned up automatically (LIFO) -``` - -### Conditional Skip with Fixtures - -**Skip tests based on runtime fixture values.** Not directly supported in pytest. - -```python -# pytest - IMPOSSIBLE: fixtures not available in skipif -@pytest.mark.skipif(config["ci"], ...) # ❌ NameError - -# ProTest - skip callable receives resolved fixtures -@session.test( - skip=lambda config: config["ci"], - skip_reason="Skip in CI", -) -def test_local_only(config: Annotated[Config, Use(config_fixture)]): - ... # Clean test body, no skip logic -``` - -*Declarative. Explicit. With full fixture access.* - --- ## Quick Start @@ -162,57 +85,6 @@ def test_answer(): protest run test_sample:session ``` -## Explicit Dependencies - -No implicit fixture resolution. You declare what you need: - -```python -from typing import Annotated -from protest import ProTestSession, Use, fixture - -session = ProTestSession() - - -@fixture() -async def database(): - db = await Database.connect() - yield db - await db.close() - -session.bind(database) # SESSION scope - - -@session.test() -async def test_create_user(db: Annotated[Database, Use(database)]): - user = await db.create_user("alice") - assert user.name == "alice" -``` - -Function-scoped fixtures use the `@fixture()` decorator: - -```python -@fixture() -def get_test_user(): - return User(name="alice") - - -@session.test() -def test_user(user: Annotated[User, Use(get_test_user)]): - assert user.name == "alice" -``` - -## Features - -- **Explicit DI** - No guessing which fixture you're using -- **Async native** - No plugin needed, just `async def` -- **Parallel execution** - Built-in with `-n 4` -- **Scoped fixtures** - `SESSION`, `SUITE`, `TEST` -- **Mix sync/async** - They just work together -- **Factory fixtures** - Return callables to create instances on-demand in tests -- **Plugin system** - Extend with custom reporters, filters -- **Last-failed mode** - Re-run only failed tests with `--lf` -- **CTRF reports** - Standardized JSON for CI/CD integration - ## Installation ProTest is not yet on PyPI. Install directly from GitHub: @@ -225,7 +97,7 @@ uv add git+https://github.com/renaudcepre/protest.git pip install git+https://github.com/renaudcepre/protest.git ``` -## CLI Usage +## CLI ```bash protest run module:session # Run tests @@ -237,270 +109,17 @@ protest run module:session --app-dir src # Look for module in src/ protest run module:session --ctrf-output r.json # CTRF report for CI/CD ``` -## API Reference - -### Core Classes - -#### `ProTestSession` - -Main entry point. Orchestrates test execution. - -```python -session = ProTestSession( - concurrency=1, # Default parallel workers -) - - -@session.test() -def test_something(): ... - - -session.add_suite(my_suite) -session.use(my_plugin) -``` - -#### `ProTestSuite` - -Groups related tests with optional max concurrency cap. - -```python -from protest import ProTestSuite - -api_suite = ProTestSuite(name="api", max_concurrency=8) - - -@api_suite.test() -async def test_endpoint(): ... - - -session.add_suite(api_suite) -``` - -### Fixtures - -Fixtures are scoped based on where they are **bound**: - -- `session.bind(fn)` - Session scope (lives entire session) -- `suite.bind(fn)` - Suite scope (lives while suite runs) -- No binding - Test scope (fresh per test) - -For fixtures that should be auto-resolved at scope start (without explicit `Use()`): - -- `session.bind(fn, autouse=True)` - Auto-resolved before any test runs -- `suite.bind(fn, autouse=True)` - Auto-resolved when suite starts - -```python -# Session-scoped fixture with teardown -@fixture() -async def database(): - db = await connect() - yield db # Teardown after yield - await db.close() - -session.bind(database) # SESSION scope - - -# Suite-scoped fixture -@fixture() -def api_config(): - return load_config() - -api_suite.bind(api_config) # SUITE scope - - -# Factory fixture - creates instances with automatic caching and teardown -@factory() -def user(name: str, role: str = "guest"): - user = User.create(name=name, role=role) - yield user - user.delete() # Teardown called for each created instance - -session.bind(user) # SESSION scope - - -# Usage: factory is injected, call it to create instances -@session.test() -async def test_multiple_users( - user_factory: Annotated[FixtureFactory[User], Use(user)] -): - alice = await user_factory(name="alice") - bob = await user_factory(name="bob", role="admin") - assert alice.name != bob.name -``` - -#### `Use` marker - -Explicitly declare dependencies with `Annotated`: - -```python -from typing import Annotated -from protest import Use - - -@session.test() -def test_with_deps( - db: Annotated[Database, Use(database)], - user: Annotated[User, Use(get_user)], -): - ... -``` - -### Parameterized Tests - -Use `ForEach` and `From` to run tests with multiple values: - -```python -from protest import ForEach, From - -CODES = ForEach([200, 201, 204]) - - -@session.test() -def test_success_codes(code: Annotated[int, From(CODES)]): - assert code in range(200, 300) # Runs 3 times -``` - -#### Parameterized Fixtures (Inversion of Control) - -In pytest, parameterized fixtures hide the iteration from the test: - -```python -# pytest - the test runs twice but you can't tell by reading it -@pytest.bind(params=["postgres", "sqlite"]) -def db(request): - return connect(request.param) - - -def test_queries(db): ... # Implicitly runs twice via fixture -``` - -In ProTest, the **test** controls the parameterization via factories: - -```python -from protest import FixtureFactory, ForEach, From, factory - -ENGINES = ForEach(["postgres", "sqlite"]) - - -@factory() -def database(engine_type: str): - db = connect(engine_type) - yield db - db.close() - -session.bind(database) - - -@session.test() -async def test_queries( - engine: Annotated[str, From(ENGINES)], - db_factory: Annotated[FixtureFactory[DB], Use(database)], -): - db = await db_factory(engine_type=engine) # Explicit wiring - assert db.is_connected() -``` - -Benefits: - -- **Visible**: `From(ENGINES)` shows the test runs multiple times -- **Explicit**: You see exactly how the fixture is configured -- **Flexible**: Other tests can use the same factory with fixed values: - -```python -@session.test() -async def test_postgres_only(db_factory: Annotated[..., Use(database)]): - db = await db_factory(engine_type="postgres") # No loop, just postgres -``` - -### Scopes (Tree-Based) - -The scope of a fixture is determined by **where** it's bound, not by the decorator: - -```python -@fixture() -def database(): ... -session.bind(database) # Session scope - lives entire session - -@fixture() -def api_client(): ... -api_suite.bind(api_client) # Suite scope - lives while suite runs - -@fixture() -def payload(): ... -# No binding = Test scope - fresh per test -``` - -| Binding | Scope | Lifecycle | -|-------------------------|------------|---------------------------------------| -| `session.bind(fn)` | Session | Created once, shared across all tests | -| `suite.bind(fn)` | Suite | Created once per suite | -| No binding | Test | Fresh instance per test | - -**Rule:** A fixture can only depend on fixtures with equal or wider scope. -Test can use Suite/Session. Suite can use Session. Session cannot use Test. - -### Built-in Fixtures - -#### `caplog` - Log Capture - -```python -from protest import caplog -from protest.entities import LogCapture - - -@session.test() -def test_logging(logs: Annotated[LogCapture, Use(caplog)]): - logging.warning("Something happened") - - assert len(logs.records) == 1 - assert "Something happened" in logs.text - assert len(logs.at_level("WARNING")) == 1 -``` - -### Plugins - -Extend ProTest by subclassing `PluginBase`: - -```python -from protest import PluginBase -from protest.entities import SessionResult, TestResult - - -class SlackNotifier(PluginBase): - def setup(self, session): - self.failures = [] - - def on_test_fail(self, result: TestResult): - self.failures.append(result.name) - - async def on_session_end(self, result: SessionResult): - if self.failures: - await send_slack_message(f"Failed: {self.failures}") - - -session.use(SlackNotifier) # Pass the class, not an instance -``` - -**Available hooks:** - -- `setup(session)` - Called when plugin is registered -- `on_collection_finish(items)` - Filter/sort collected tests -- `on_session_start()` - Before any test runs -- `on_session_end(result)` - Tests done, async handlers can start -- `on_session_complete(result)` - All async handlers finished -- `on_suite_start(info)` / `on_suite_end(result)` -- `on_test_pass(result)` / `on_test_fail(result)` / `on_test_skip(result)` - -### Cache Plugin (Built-in) - -Re-run only failed tests: - -```bash -protest run module:session --lf # Run last-failed tests -protest run module:session --cache-clear # Clear cache -``` +## Features -Cache stored in `.protest/cache.json`. +- **Explicit DI** - No guessing which fixture you're using +- **Async native** - No plugin needed, just `async def` +- **Parallel execution** - Built-in with `-n 4` +- **Scoped fixtures** - `SESSION`, `SUITE`, `TEST` +- **Mix sync/async** - They just work together +- **Factory fixtures** - Callables to create instances on-demand +- **Plugin system** - Custom reporters, filters +- **Last-failed mode** - Re-run only failed tests with `--lf` +- **CTRF reports** - Standardized JSON for CI/CD integration ## Why Not pytest? @@ -515,6 +134,10 @@ Cache stored in `.protest/cache.json`. pytest has a large ecosystem and extensive community. ProTest is an alternative if you prefer FastAPI-style explicit dependencies and native async in your tests. +## Documentation + +Full API reference, guides, and examples: [renaudcepre.github.io/protest](https://renaudcepre.github.io/protest/) + ## License MIT diff --git a/docs/architecture/fixture-scoping-design.md b/docs/architecture/fixture-scoping-design.md deleted file mode 100644 index 374cbc3..0000000 --- a/docs/architecture/fixture-scoping-design.md +++ /dev/null @@ -1,273 +0,0 @@ -# Fixture Scoping Design Decision - -## Contexte - -ProTest est un framework de test async-first avec injection de dépendances explicite. Les fixtures peuvent avoir trois scopes : - -- **SESSION** : Une instance pour toute la session de tests -- **SUITE** : Une instance par suite de tests -- **TEST** : Une instance fraîche par test (défaut) - -Le défi architectural est de permettre l'organisation des fixtures en modules séparés tout en gardant une API explicite et sans ambiguïté. - ---- - -## Approche 1 : Tree-Based (rejetée) - -### Principe - -Le scope est déterminé par **où** on décore la fixture : - -```python -# fixtures.py -from myproject.session import session # Import nécessaire - -@session.bind() -def database(): - yield connect() - -@api_suite.bind() -def client(): - return Client() -``` - -### Avantages - -- API explicite : une action = scope + binding -- Impossible d'avoir une fixture "orpheline" -- Mental model simple - -### Problème bloquant : Imports circulaires - -``` -fixtures.py ←──imports── session.py - │ │ - └────imports───────────────┘ -``` - -Pour décorer avec `@session.bind()`, le fichier fixtures.py doit importer `session`. Mais session.py doit importer les fixtures pour les utiliser dans les tests. - -**Verdict** : Incompatible avec une architecture modulaire (fichiers > 400 lignes sinon). - ---- - -## Approche 2 : Mark & Collect (rejetée) - -### Principe - -Séparer la déclaration du scope et le binding : - -```python -# fixtures.py (pas d'import de session) -from protest import fixture, FixtureScope - -@fixture(scope=FixtureScope.SESSION) -def database(): - yield connect() - -@fixture(scope=FixtureScope.SUITE) -def client(): - return Client() -``` - -```python -# session.py -from .fixtures import database, client - -session = ProTestSession() -api_suite = ProTestSuite("API") - -session.use_fixtures([database]) -api_suite.use_fixtures([client]) -``` - -### Avantages - -- Pas de cycle d'import -- Fixtures dans des modules séparés - -### Problèmes identifiés - -| Problème | Description | -|----------|-------------| -| **Deux étapes requises** | Décorateur (`scope=X`) + binding (`use_fixtures`). Redondant et source d'erreurs. | -| **Incohérence possible** | `@fixture(scope=SUITE)` + `session.use_fixtures()` = mismatch silencieux ou erreur ? | -| **Fixtures orphelines** | Une fixture `scope=SUITE` sans binding a un comportement implicite surprenant (scope "flottant" = une instance par suite appelante). | -| **Double source de vérité** | Le scope est déclaré au décorateur ET vérifié au binding. Lequel fait foi ? | - -**Verdict** : Résout les imports mais introduit de l'ambiguïté et des états incohérents. - ---- - -## Approche 3 : Scope au Binding (retenue) - -### Principe - -Le décorateur marque une fonction comme fixture, **sans déclarer de scope**. Le scope est déterminé uniquement par le binding. - -```python -# fixtures.py (pas d'import de session, pas de scope) -from protest import fixture - -@fixture() -def database(): - yield connect() - -@fixture() -def client(): - return Client() - -@fixture() -def temp_file(): - yield "/tmp/test" -``` - -```python -# session.py -from protest import ProTestSession, ProTestSuite -from .fixtures import database, client - -session = ProTestSession() -api_suite = ProTestSuite("API") - -session.bind(database) # database → SESSION scope -api_suite.bind(client) # client → SUITE scope -# temp_file non bindée # temp_file → TEST scope (défaut) - -session.add_suite(api_suite) -``` - -### Utilisation dans les tests - -```python -@api_suite.test() -def test_example( - db: Annotated[Connection, Use(database)], # SESSION (bindé à session) - c: Annotated[Client, Use(client)], # SUITE (bindé à api_suite) - tmp: Annotated[str, Use(temp_file)], # TEST (pas bindé = défaut) -): - pass -``` - -### Règles de résolution - -1. **Fixture bindée** → utilise le scope du binding -2. **Fixture non bindée** → TEST scope par défaut -3. **Double binding** → Erreur explicite (`AlreadyRegisteredError`) - -### Avantages - -| Aspect | Bénéfice | -|--------|----------| -| **Pas de cycle d'import** | fixtures.py n'importe pas session.py | -| **Source de vérité unique** | Le scope est défini au binding, pas au décorateur | -| **Pas d'orphelins ambigus** | Non bindé = TEST scope (comportement sain et prévisible) | -| **API familière** | `session.bind(fn)` ressemble à `@session.bind()` | -| **Simplicité du décorateur** | `@fixture()` fait une seule chose : marquer comme fixture | -| **Erreurs explicites** | Double binding = erreur, pas de comportement implicite | - -### Inconvénient accepté - -On ne peut pas déterminer le scope d'une fixture en regardant uniquement son décorateur. Il faut regarder où elle est bindée. - -**Justification** : Le scope d'une fixture dépend de son contexte d'utilisation, pas de sa définition. Une même fixture pourrait théoriquement être SESSION dans un projet et SUITE dans un autre. Le binding rend cette dépendance explicite. - ---- - -## Migration depuis Mark & Collect - -### Avant (Mark & Collect) - -```python -@fixture(scope=FixtureScope.SESSION, tags=["database"]) -def database(): - yield connect() - -session.use_fixtures([database]) -``` - -### Après (Scope au Binding) - -```python -@fixture(tags=["database"]) -def database(): - yield connect() - -session.bind(database) -``` - -### Changements - -1. Retirer `scope=` du décorateur `@fixture()` -2. Remplacer `session.use_fixtures([fn])` par `session.bind(fn)` -3. Remplacer `suite.use_fixtures([fn])` par `suite.bind(fn)` - ---- - -## Diagramme de décision - -``` - @fixture() - │ - ▼ - ┌─────────────────────┐ - │ Fixture marquée │ - │ (pas de scope) │ - └─────────────────────┘ - │ - ┌─────────────┼─────────────┐ - ▼ ▼ ▼ - session.bind() suite.bind() (rien) - │ │ │ - ▼ ▼ ▼ - SESSION SUITE TEST - scope scope scope -``` - ---- - -## Validation du scope des dépendances - -Les règles de scope restent inchangées : - -- SESSION peut dépendre de : SESSION uniquement -- SUITE peut dépendre de : SESSION, SUITE (même suite ou parent) -- TEST peut dépendre de : tout - -```python -@fixture() -def config(): - return {} - -@fixture() -def database(cfg: Annotated[dict, Use(config)]): - yield connect(cfg) - -session.bind(config) # SESSION -session.bind(database) # SESSION - OK (dépend de SESSION) -``` - -```python -session.bind(config) # SESSION -api_suite.bind(database) # SUITE dépend de SESSION → OK -``` - -```python -api_suite.bind(config) # SUITE -session.bind(database) # SESSION dépend de SUITE → ERREUR -``` - ---- - -## Conclusion - -L'approche "Scope au Binding" combine les avantages des deux approches précédentes : - -- **De Tree-Based** : une seule action pour définir le scope (`session.bind(fn)`) -- **De Mark & Collect** : pas de cycles d'import - -Elle élimine les inconvénients : - -- **Pas de redondance** : le scope n'est déclaré qu'une fois -- **Pas d'ambiguïté** : le binding EST le scope -- **Pas d'orphelins** : non bindé = TEST scope explicite \ No newline at end of file