- Add 9 Chainsaw E2E test suites covering Keystone reconciler scenarios:
basic-deployment, missing-secret, fernet-rotation, scale,
deletion-cleanup, policy-overrides, middleware-config,
brownfield-database, and image-upgrade
- Validate full reconciliation cycles with JMESPath condition assertions
on all 5 sub-conditions (DatabaseReady, SecretsReady, FernetKeysReady,
BootstrapReady, DeploymentReady) and aggregate Ready condition
- Verify owned resource creation (Deployment, Service, CronJob, Secret,
RBAC, PushSecret, ConfigMap) and proper cleanup on CR deletion
- Test day-2 operations: replica scaling, image upgrades with rolling
restarts, and fernet key rotation with annotation hash changes
- Cover error recovery path where missing secrets cause degraded state
followed by reconciliation to Ready after late secret creation
- Test brownfield database mode with pre-existing credentials and
policy overrides via external ConfigMap references
- Validate middleware configuration propagation through the pipeline
config chain
- Add comprehensive reference documentation for all E2E test suites
in docs/reference/keystone-e2e-tests.md
- Update VitePress sidebar config and keystone-crd reference to
cross-link the new E2E test documentation
- Add Chainsaw E2E test structure pattern to .planwerk/patterns for
consistent JMESPath condition assertion conventions
AI-assisted: Claude Code
On-behalf-of: @SAP christian.berendt@sap.com
Signed-off-by: Christian Berendt <berendt@23technologies.cloud>
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
berendt
force-pushed
the
feature/CC-0016
branch
from
Size: 🏗️ large
Category: infrastructure
Priority: medium
Feature ID: CC-0016 (S016)
Open Questions
No open questions — both decisions resolved by user:
missing-secret/isolation → Use uniquesecretRefnames per test CRparallel: 4→ Unique CR name per test directorySummary
Create 9 Chainsaw E2E test suites under
tests/e2e/keystone/validating the Keystone operator's full lifecycle on a real cluster. Each suite targets a specific operational scenario: happy-path deployment, secret dependency recovery, Fernet key rotation, replica scaling, garbage collection on deletion, policy override injection, middleware pipeline customization, brownfield (unmanaged) database mode, and rolling image upgrades. Together they cover the operator's condition progression (SecretsReady→DatabaseReady→FernetKeysReady→DeploymentReady→BootstrapReady→Ready), owned resource management, and resilience to operational changes.Scope
Included:
basic-deployment/— Apply valid Keystone CR (managed mode withclusterRef), assert all 5 sub-conditions progress toTrueand aggregateReady=True(reason:AllReady), verify Deployment{name}-apiwithavailableReplicas > 0, Service{name}-apion port 5000, immutable ConfigMap{name}-config-{hash}, CronJob{name}-fernet-rotate, fernet Secret{name}-fernet-keys, RBAC resources (ServiceAccount/Role/RoleBinding{name}-fernet-rotate), PushSecret{name}-fernet-keys-backup. Singlescriptstep withcurl http://{name}-api.openstack.svc:5000/v3to verify functional API response.missing-secret/— Apply Keystone CR referencing non-existent Secret names (unique per test, e.g.missing-secret-keystone-db), assertSecretsReady=False(reason:WaitingForDBCredentialsorWaitingForAdminCredentials), create prerequisite Secrets, assert recovery toReady=Truefernet-rotation/— Verify CronJob{name}-fernet-rotateschedule matchesspec.fernet.rotationSchedule, trigger manual rotation viakubectl create job --from=cronjob/{name}-fernet-rotate, assert{name}-fernet-keysSecret.datachanges, assert Deployment pod-template annotationkeystone.c5c3.io/fernet-keys-hashupdates (rolling restart)scale/— Apply Keystone CR withreplicas: 3, patch toreplicas: 5, assertDeployment.spec.replicasandstatus.availableReplicasupdate, patch toreplicas: 2, assert scale-downdeletion-cleanup/— Deploy Keystone CR, wait forReady=True, delete CR, assert all owned resources return NotFound viaerrorchecks: Deployment, Service, ConfigMap, CronJob, Jobs (db-sync, bootstrap), Secret (fernet-keys), ServiceAccount, Role, RoleBinding, PushSecretpolicy-overrides/— Apply Keystone CR withpolicyOverrides.configMapRef, assert generated ConfigMap containspolicy.yamldata key, assertkeystone.confcontains[oslo_policy]section withpolicy_file = /etc/keystone/policy.yamlmiddleware-config/— Apply Keystone CR with custommiddlewareentries (name, filterFactory, position), assertapi-paste.inikey in generated ConfigMap contains the modified pipeline filter referencesbrownfield-database/— Apply Keystone CR with explicitdatabase.host: openstack-db.openstack.svc.cluster.localanddatabase.port: 3306(noclusterRef), assert NOdatabases.k8s.mariadb.comorusers.k8s.mariadb.comorgrants.k8s.mariadb.comCRs are created (viaerrorchecks), assertkeystone.conf[database].connectioncontains the explicit hostimage-upgrade/— Apply Keystone CR, wait forReady=True, patchspec.image.tag, assert Deployment container image updates, assertReady=Truemaintained after rollout completesExcluded:
invalid-cr/in CC-0012)Visualization
flowchart TD subgraph E2E["tests/e2e/keystone/"] BD["basic-deployment/"] MS["missing-secret/"] FR["fernet-rotation/"] SC["scale/"] DC["deletion-cleanup/"] PO["policy-overrides/"] MC["middleware-config/"] BF["brownfield-database/"] IU["image-upgrade/"] end subgraph Conditions["Condition Coverage"] SR["SecretsReady"] DR["DatabaseReady"] FKR["FernetKeysReady"] DPR["DeploymentReady"] BR["BootstrapReady"] RDY["Ready"] end BD --> SR & DR & FKR & DPR & BR & RDY MS --> SR FR --> FKR SC --> DPR DC --> RDY PO --> DR & DPR MC --> DPR BF --> DR & DPR IU --> DPR & RDYsequenceDiagram participant CH as Chainsaw participant K8s as Kubernetes API participant KR as Keystone Reconciler participant OW as Owned Resources Note over CH: basic-deployment CH->>K8s: apply Keystone CR + prerequisite Secrets K8s->>KR: reconcile KR->>OW: create fernet Secret, CronJob, RBAC, ConfigMap, MariaDB CRs, db-sync Job, Deployment, Service, bootstrap Job, PushSecret CH->>K8s: assert conditions SecretsReady...Ready = True CH->>K8s: assert Deployment availableReplicas > 0 CH->>K8s: script curl /v3 Note over CH: missing-secret CH->>K8s: apply Keystone CR with unique secretRef names CH->>K8s: assert SecretsReady = False CH->>K8s: create prerequisite Secrets CH->>K8s: assert Ready = True Note over CH: fernet-rotation CH->>K8s: apply Keystone CR, wait Ready CH->>K8s: record fernet Secret .data hash CH->>K8s: script kubectl create job --from=cronjob CH->>K8s: assert fernet Secret .data changed CH->>K8s: assert Deployment annotation changed Note over CH: deletion-cleanup CH->>K8s: delete Keystone CR CH->>K8s: error assert Deployment not found CH->>K8s: error assert Service not found CH->>K8s: error assert CronJob not foundstateDiagram-v2 [*] --> SecretsReady: ESO Secrets synced SecretsReady --> DatabaseReady: MariaDB CRs + db_sync Job DatabaseReady --> FernetKeysReady: Fernet Secret + CronJob + PushSecret FernetKeysReady --> DeploymentReady: Deployment available + endpoint set DeploymentReady --> BootstrapReady: Bootstrap Job complete BootstrapReady --> Ready: All conditions True state "missing-secret test" as ms SecretsReady --> ms: SecretsReady=False path state "fernet-rotation test" as fr FernetKeysReady --> fr: Key rotation path state "scale test" as sc DeploymentReady --> sc: Replica changesKey Components
basic-deployment/chainsaw-test.yaml+ fixture CR + prerequisite Secrets — Happy-path test asserting full condition progression (SecretsReady→DatabaseReady→FernetKeysReady→DeploymentReady→BootstrapReady→Ready=Truewith reasonAllReady), resource existence checks (Deployment{name}-api, Service, ConfigMap, CronJob, fernet Secret, RBAC, PushSecret), and ascriptstep withcurlto verify HTTP response from/v3missing-secret/chainsaw-test.yaml+ fixture CR + late-created Secrets — CR references unique Secret names (e.g.missing-secret-keystone-db,missing-secret-keystone-admin). TestsSecretsReady=Falsewhen referenced Secrets don't exist, asserts reason isWaitingForDBCredentialsorWaitingForAdminCredentials, then creates Secrets and asserts recovery toReady=Truefernet-rotation/chainsaw-test.yaml+ fixture CR — Verifies CronJobschedulefield matches spec, triggers rotation viakubectl create job --from=cronjob/{name}-fernet-rotatein ascriptstep, asserts{name}-fernet-keysSecret.datachanges, asserts Deployment pod-template annotationkeystone.c5c3.io/fernet-keys-hashvalue changes (triggering rolling restart)scale/chainsaw-test.yaml+ fixture CR + patch files — Three-step test (3→5→2) using Chainsawpatchsteps onspec.replicas, assertingDeployment.spec.replicasandstatus.availableReplicasat each stepdeletion-cleanup/chainsaw-test.yaml+ fixture CR — Deploys Keystone CR, waits forReady=True, deletes CR, asserts all owned resources return NotFound via Chainsawerrorchecks: Deployment, Service, ConfigMap, CronJob, Jobs (db-sync, bootstrap), Secret (fernet-keys), ServiceAccount, Role, RoleBinding, PushSecretpolicy-overrides/chainsaw-test.yaml+ fixture CR + policy ConfigMap — Applies Keystone CR withpolicyOverrides.configMapRef, asserts generated ConfigMap containspolicy.yamldata key andkeystone.confcontains[oslo_policy]section withpolicy_filemiddleware-config/chainsaw-test.yaml+ fixture CR — Applies Keystone CR with custommiddlewareentries (specifyingname,filterFactory,position), assertsapi-paste.iniin generated ConfigMap contains the modified pipeline filter referencesbrownfield-database/chainsaw-test.yaml+ fixture CR — Applies Keystone CR withdatabase.host/database.port(noclusterRef), asserts NOdatabases.k8s.mariadb.com,users.k8s.mariadb.com, orgrants.k8s.mariadb.comCRs exist viaerrorchecks, assertskeystone.conf[database].connectioncontains the explicit host. Uses the existingopenstack-dbMariaDB instance.image-upgrade/chainsaw-test.yaml+ fixture CR + patch file — Applies Keystone CR, waits forReady=True, patchesspec.image.tagvia Chainsawpatchstep, asserts Deployment container image updates, assertsReady=Truemaintained after rollout completeskeystone-basic,keystone-scale). Each test creates its own prerequisite Secrets with test-unique names. Brownfield tests usehost/port; managed tests useclusterRefpointing toopenstack-db/openstack-memcached. Assertions follow existing patterns frominfra-stack-healthandinvalid-cr: CEL expressions with backtick-quoted numerics (`0`), JMESPath condition filters (conditions[?type == 'Ready']), and extendedasserttimeout (5m) for full reconciliation cycles.Note on condition order: Research confirms the sub-reconciler execution order is
reconcileSecrets→reconcileDatabase→reconcileFernetKeys→reconcileConfig→reconcileDeployment→reconcileBootstrap(perkeystone_controller.gosubConditionTypes). The user's original elaboration hadFernetKeysReadybeforeDatabaseReady— corrected above to match the actual code.