feat: add SQLite storage as single source of truth

[jimisola]

Improvements

This PR replaces the hand-built, multi-stage intermediate data pipeline with an in-memory SQLite database as the single source of truth.

Complexity reduction

Area	Before	After
Data pipeline stages	4 (RawDataset → CombinedRawDataset → CombinedIndexedDataset → StatisticsContainer)	2 (RawDataset → SQLite INSERT → query)
Hand-built index dicts	5 (svcs_from_req, mvrs_from_svc, reqs_from_urn, etc.)	0 — SQL FK relationships
Filtering implementation	390-line imperative IndexedDatasetFilterProcessor with recursive cleanup	SQL DELETE with ON DELETE CASCADE
EL transformers	2 empty subclasses + base class evaluating per-item in Python	1 compiler emitting SQL WHERE clauses
Status table columns	13 (5 sub-columns × 2 groups + 3) with 97-line merged header renderer	5 flat columns with inline formatted cells
Statistics computation	313-line StatisticsGenerator with manual aggregation loops	SQL queries via StatisticsService

What was removed (1,181 lines)

• StatisticsContainer / TotalStatisticsItem (105 lines)
• StatisticsGenerator (313 lines)
• CombinedIndexedDatasetGenerator (302 lines)
• IndexedDatasetFilterProcessor (390 lines)
• CombinedIndexedDataset (53 lines)
• Per-item EL transformers (18 lines)

What replaced it (1,567 lines)

• Storage layer (1,059 lines): schema, database wrapper, populator, SQL filter processor, EL-to-SQL compiler, repository, authorizer, pipeline
• Services layer (508 lines): StatisticsService, ExportService

The net increase of ~386 lines trades imperative index-maintenance code for a declarative storage layer with FK constraints, CASCADE rules, ACID semantics, and a security authorizer.

---

Summary

Replace intermediate data structures (CombinedIndexedDataset, StatisticsContainer, StatisticsGenerator) with an in-memory SQLite database as the single source of truth. All commands (status, export, report) now query the database through a RequirementsRepository and service layer instead of traversing in-memory dicts.

Phases completed in this PR

• Phase 1-2: SQLite infrastructure — schema (13 tables with CHECK constraints, FK indexes), RequirementsDatabase, DatabasePopulator, authorizer (security sandbox)
• Phase 3: EL-to-SQL compiler and DatabaseFilterProcessor for applying requirement/SVC filters via SQL DELETE
• Phase 4: Service layer, repository, and command rewrites
  • RequirementsRepository — data access layer with all entity queries and test result resolution
  • StatisticsService — replaces StatisticsGenerator + StatisticsContainer with frozen TestStats/RequirementStatus/TotalStats dataclasses
  • ExportService — replaces GenerateJsonCommand inline logic with --req-ids/--svc-ids filtering
  • build_database() — pipeline helper: Location → parse → populate DB → filter → validate
  • LifecycleValidator and GroupByOrganizor migrated from CombinedIndexedDataset to RequirementsRepository
  • Multi-pass DB population to handle cross-URN FK constraints
• JSON Schemas: status_output.schema.json and export_output.schema.json for output validation
• Status table redesign: Collapsed 13-column table (5 sub-columns per test group) into 5 columns with compact inline formatting — positionally-aligned counts with colored numbers, dim dashes for zeros, and color-coded legend

Status table (before → after)

Before: 13 columns with complex merged header spanning T/P/F/S/M sub-columns for each test group.

After: 5 clean columns — each test cell shows inline counts with colors (green=passed, red=failed, yellow=skipped, orange=missing):

Key design decisions

• Pipeline: Location → CombinedRawDatasetsGenerator(database=db) → DatabaseFilterProcessor → RequirementsRepository → commands
• Old code (StatisticsGenerator, StatisticsContainer, CombinedIndexedDataset, etc.) kept for Phase 5 cleanup
• Report uses _TotalStatsTemplateAdapter to bridge new attribute names to existing Jinja2 templates
• References in reports are now sorted for deterministic output

Closes

• Closes investigate: replace intermediate data structures with in-memory SQLite as single source of truth #313 — SQLite as single source of truth
• Closes investigate: rename/refactor StatisticsContainer and related models for clarity #312 — StatisticsContainer renamed/refactored into StatisticsService + clean dataclasses
• Closes feat: define JSON Schema for export and status command output #315 — JSON Schema for export and status output
• Closes investigate: JSON export strategy with SQLite backend #317 — JSON export strategy with SQLite backend (Option 3: query → dict → validate against schema)
• Closes epic: separate internal data layer from user-facing output contracts #318 — Separate internal data layer from user-facing output contracts (services layer decouples all output formats from internal models)

Related

• epic: reqstool architecture evolution #316 — epic: reqstool architecture evolution (Phase 1–2 complete; Phase 3–4 remain)

Test plan

• All 300 unit tests pass
• Lint clean (flake8)
• Status output verified for test_standard, test_basic fixtures, and reqstool-demo
• Report output matches main (only difference: sorted reference ordering — improvement)
• Status JSON validates against status_output.schema.json
• Export JSON validates against export_output.schema.json
• JSON output unchanged by status table redesign (separate code path)
• Verify against reqstool-demo (requires ./mvnw verify in sibling project)
• Manual CLI testing per TEST_MATRIX.md scenarios

🤖 Generated with Claude Code

[jimisola]

[jimisola]

@Jonas-Werne @lfvdavid It runs, regression tests passes etc, so please try locally.

[jimisola]

Regression Test Results

Full regression comparison of main vs feat/313-sqlite-storage across 3 datasets: test_standard/baseline/ms-001, test_basic/baseline/ms-101, and reqstool-demo.

Reports (asciidoc + markdown) — 6/6 cosmetic only

All report diffs are cosmetic:

• Extra blank line after title
• Trailing whitespace removed from header lines
• Sorted reference ordering (e.g. ext-001:REQ_ext001_100, ms-001:REQ_010 instead of ms-001:REQ_010, ext-001:REQ_ext001_100) — alphabetical sort, an improvement

No data value changes in any report output.

Export JSON — intentional schema changes

All diffs are the new schema per #315:

• metadata wrapper with renamed keys (initial_urn, import_graph)
• urn/id split (was composite "id": "ms-001:REQ_010")
• implementation → implementation_type
• annotations_impls/annotations_tests → annotations.implementations/annotations.tests
• automated_test_result → test_results
• Internal lookup maps removed (reqs_from_urn, svcs_from_req, mvrs_from_svc, etc.)
• Clean enum values ("effective" instead of jsonpickle format)

Status JSON — intentional schema changes

Same schema redesign as export: cleaner naming, nested totals structure, metadata section.

Bugs found and fixed

1. "SVCs missing tests/MVRs" regression — TotalStats.missing_automated_tests and missing_manual_tests were never aggregated from per-requirement stats, always reporting 0. Fixed by accumulating from each requirement's TestStats in _calculate(). Counts now match main (e.g. test_standard: 2/2).

2. FK crash on test_errors dataset — Dangling FK references (SVC→non-existent REQ, MVR→non-existent SVC, annotations→non-existent targets) caused sqlite3.IntegrityError crash. Fixed by catching IntegrityError in insert_svc, insert_mvr, insert_annotation_impl, and insert_annotation_test with warning-level logging. Semantic validation now runs and reports all errors as before.

Stale baselines removed

The baselines/ directory was captured from pre-Pydantic-v2 main (784f77b) and was outdated. Removed after verifying regression directly against current main.

[Jonas-Werne]

I removed the commented out import from the demo project and got this and it did not get a enumerated value.

But more importantly, with the current representation you can't see the ID of the requirement this will not work.

I like the enumerations but they can't replace the ID.

We could combine the URN + ID column to contain the full ID

reqstool-demo:REQ_001, ext-001:REQ_001 etc, then we could change the ID column to say STATUS that include the Enumerated status

[jimisola]

I removed the commented out import from the demo project and got this and it did not get a enumerated value.

What enumerated value are you referring to?

But more importantly, with the current representation you can't see the ID of the requirement this will not work.

REQ_PASS, REQ_MANUAL_FAIL etc are the IDs of those requirements as per: https://github.com/reqstool/reqstool-demo/blob/main/docs/reqstool/requirements.yml

What do you mean that you can't see the ID?

[jimisola]

We should add @requirements that are only implemented once to have that case as well:

https://github.com/reqstool/reqstool-demo/blob/5c77cbb39f442955f25d2e7023c442024743e72c/src/main/java/io/github/reqstool/example/demo/RequirementsExample.java#L5-L8

Or rather, remove the class level annotations except for one requirement that we name accordingly, e.g. REQ_IMPLEMENTED_2_TIMES"

[Jonas-Werne]

I removed the commented out import from the demo project and got this and it did not get a enumerated value.

What enumerated value are you referring to?

But more importantly, with the current representation you can't see the ID of the requirement this will not work.

REQ_PASS, REQ_MANUAL_FAIL etc are the IDs of those requirements as per: https://github.com/reqstool/reqstool-demo/blob/main/docs/reqstool/requirements.yml

What do you mean that you can't see the ID?

My mistake, I confused the ID to be some status of the requirement. Like REQ_PASS would mean everything was done and REQ_NOT_IMPLEMENTED meant that you have not annotated the code.

[jimisola]

I removed the commented out import from the demo project and got this and it did not get a enumerated value.

What enumerated value are you referring to?

But more importantly, with the current representation you can't see the ID of the requirement this will not work.

REQ_PASS, REQ_MANUAL_FAIL etc are the IDs of those requirements as per: https://github.com/reqstool/reqstool-demo/blob/main/docs/reqstool/requirements.yml
What do you mean that you can't see the ID?

My mistake, I confused the ID to be some status of the requirement. Like REQ_PASS would mean everything was done and REQ_NOT_IMPLEMENTED meant that you have not annotated the code.

No, worries. I appreciate you taking a look. I though that the REQ_nnn were not self-explanatory as the new IDs are.

[Jonas-Werne]

I think @lfvdavid has the best data to regression test this. Maybe we should generate a more complex data in the demo project.

All levels microservice, system, external and include filtering of requirements and also some MVR

No valid change request

[jimisola]

I think @lfvdavid has the best data to regression test this. Maybe we should generate a more complex data in the demo project.

All levels microservice, system, external and include filtering of requirements and also some MVR

Ideally, I don't want the demo to be too complex. It shall just demonstrate the basics for new users. What do you think?

I rather have a more complex example directly in reqstool-client then. And we still need to dog food reqstool-client and others with reqstool.