CORS-4336: Add CI jobs for AWS European Sovereign Cloud (EUSC)#75568
CORS-4336: Add CI jobs for AWS European Sovereign Cloud (EUSC)#75568liweinan wants to merge 12 commits intoopenshift:mainfrom
Conversation
|
@liweinan: This pull request references CORS-4336 which is a valid jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@liweinan, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/cluster-profiles/cluster-profiles-config.yaml
Outdated
Show resolved
Hide resolved
...ipi/private/provision/cucushift-installer-rehearse-aws-eusc-ipi-private-provision-chain.yaml
Outdated
Show resolved
Hide resolved
...ipi/private/provision/cucushift-installer-rehearse-aws-eusc-ipi-private-provision-chain.yaml
Outdated
Show resolved
Hide resolved
...ipi/private/provision/cucushift-installer-rehearse-aws-eusc-ipi-private-provision-chain.yaml
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/ipi/conf/aws/eusc-ami/ipi-conf-aws-eusc-ami-commands.sh
Outdated
Show resolved
Hide resolved
|
@liweinan as we discussed offline, for the new partition we need three types of cluster:
|
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
|
@yunjiang29 Thanks for the review! I'll refactor this PR today. |
24fed80 to
de00d69
Compare
|
@liweinan, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@yunjiang29 Thanks for the detailed review! I'll update the PR recordingly. |
|
@liweinan, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
Address yunfei's review comments on PR openshift#75568: 1. Job naming convention: - Rename jobs from -f60 to -f7 suffix (non-destructive tests) - Update cron schedule to standard f7 pattern: 7,14,23,30 2. Private cluster configuration: - Add complete private cluster setup with bastion host - Add VPC, security groups, and proxy configuration - Set PUBLISH=Internal for private cluster access - Add minimal IAM permission provisioning - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision 3. AMI configuration fix: - Replace deprecated compute.platform.aws.amiID field - Use platform.aws.defaultMachinePlatform.amiID instead
4b73bfe to
7f83d83
Compare
|
@liweinan, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
1. Job naming convention:
- Rename jobs from -f60 to -f7 suffix (non-destructive tests)
- Update cron schedule to standard f7 pattern: 7,14,23,30
2. Private cluster configuration:
- Add complete private cluster setup with bastion host
- Add VPC, security groups, and proxy configuration
- Set PUBLISH=Internal for private cluster access
- Add minimal IAM permission provisioning
- Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision
3. AMI configuration fix:
- Replace deprecated compute.platform.aws.amiID field
- Use platform.aws.defaultMachinePlatform.amiID instead
4. Generalize step registry components for reusability:
- Enhance ipi-conf-aws-custom-endpoints to support multiple AWS partitions
* Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com)
* Support amazonaws.eu for EUSC, amazonaws.com.cn for China
* Allow full URLs for maximum flexibility
- Make ipi-conf-aws-eusc-ami more generic
* Support AWS_CUSTOM_AMI_ID for general use
* Maintain AWS_EUSC_AMI_ID for backward compatibility
* Can be used for EUSC, China, GovCloud, or custom AMI scenarios
- Use generic steps in EUSC provision chain with partition-specific config
- Remove obsolete ipi-conf-aws-eusc-endpoints (replaced by generic version)
7f83d83 to
55daf83
Compare
|
@liweinan, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
...erator/step-registry/ipi/conf/aws/custom-endpoints/ipi-conf-aws-custom-endpoints-commands.sh
Outdated
Show resolved
Hide resolved
1. Job naming convention:
- Rename jobs from -f60 to -f7 suffix (non-destructive tests)
- Update cron schedule to standard f7 pattern: 7,14,23,30
2. Private cluster configuration:
- Add complete private cluster setup with bastion host
- Add VPC, security groups, and proxy configuration
- Set PUBLISH=Internal for private cluster access
- Add minimal IAM permission provisioning
- Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision
3. Generalize step registry components for maximum reusability:
a) Enhance ipi-conf-aws-custom-endpoints for all AWS partitions:
- Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com)
- Support amazonaws.eu (EUSC), amazonaws.com.cn (China)
- Allow full URLs for maximum flexibility
- Remove obsolete ipi-conf-aws-eusc-endpoints step
b) Extend ipi-conf-aws to support custom AMI configuration:
- Add AWS_AMI_ID env var for custom RHCOS AMI
- Useful for EUSC, China, GovCloud, or any partition without public AMIs
- Fix deprecated amiID field -> defaultMachinePlatform.amiID
- Auto-detection still works for C2S/SC2S
- Remove obsolete ipi-conf-aws-eusc-ami step
c) EUSC provision chain now uses only generic steps with env config
This refactoring reduces code duplication (net -59 lines) and makes step
components reusable across all AWS partitions.
55daf83 to
c6c4827
Compare
|
@liweinan, Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
Relative PRs merged: #75441 / openshift/ci-tools#4973 |
Update generated Prow job configurations after rebasing to the latest origin/main. Changes include: - Updated cluster assignments to match current build cluster distribution - EUSC jobs properly integrated with latest job generation logic
17c85bd to
c96110f
Compare
Delete 4 EUSC-specific workflows and 2 provision chains, replacing them with standard AWS workflows. This reduces maintenance burden and ensures consistency with standard AWS job configurations. Changes: - Delete cucushift-installer-rehearse-aws-eusc-ipi workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi-private workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi-private-sts workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi provision chain - Delete cucushift-installer-rehearse-aws-eusc-ipi-private provision chain Modified 5 jobs to use standard AWS workflows: - aws-eusc-ipi-fips-f7 → cucushift-installer-rehearse-aws-ipi - aws-eusc-ipi-f28-destructive → cucushift-installer-rehearse-aws-ipi - aws-eusc-ipi-private-sts-fips-f7 → aws-ipi-private-cco-manual-security-token-service - aws-eusc-ipi-private-mini-perm-f28 → cucushift-installer-rehearse-aws-ipi-private - aws-eusc-ipi-disc-priv-f28 → cucushift-installer-rehearse-aws-ipi-disconnected-private All modified jobs now include: - cluster_profile: aws-eusc (handles region and AMI configuration) - COMPUTE_NODE_TYPE: m5.xlarge - CONTROL_PLANE_INSTANCE_TYPE: m6i.xlarge Preserved for further discussion: - cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private-kms (unique combination not available in standard AWS workflows) Result: -300 lines, 100% workflow reuse for modified jobs
aa44a64 to
987a500
Compare
- Delete last EUSC-specific workflow: cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private-kms - Delete associated provision chain - Update aws-eusc-ipi-disc-priv-kms-f7 job to use standard cucushift-installer-rehearse-aws-ipi-disconnected-private workflow - Add COMPUTE_NODE_TYPE and CONTROL_PLANE_INSTANCE_TYPE env vars to the job All EUSC jobs now use standard AWS workflows with cluster_profile: aws-eusc. This completes the refactoring based on review feedback.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: liweinan The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Two critical bug fixes in ipi-conf-aws-commands.sh:
1. Fix CONTROL_PLANE_AMI being unconditionally overwritten
- Before: Always fetched from GitHub in C2S/SC2S environments
- After: Only auto-detect if user hasn't provided CONTROL_PLANE_AMI
- Impact: Users can now override AMI for control plane nodes
2. Fix COMPUTE_AMI being unconditionally overwritten
- Before: COMPUTE_AMI="${CONTROL_PLANE_AMI}" (always overwrites)
- After: COMPUTE_AMI="${COMPUTE_AMI:-${CONTROL_PLANE_AMI}}" (respects user value)
- Impact: Users can now specify different AMIs for compute nodes
Both fixes are 100% backward compatible with existing jobs.
All current C2S/SC2S jobs don't set these env vars, so behavior unchanged.
|
Here is the list of PRs to support EUSC: openshift/installer#10303 (see comment) |
ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-commands.sh
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly.yaml
Outdated
Show resolved
Hide resolved
...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly.yaml
Show resolved
Hide resolved
|
@yunjiang29 Thanks for the detailed review! I'll update the PR accordingly. |
Changes per yunjiang29's review comments:
1. Remove all 6 EUSC jobs from amd64-nightly.yaml
- All EUSC jobs now run against multi-nightly payload only
- ARM for non-destructive (f7), AMD for destructive (f28)
2. Fix ipi-conf-aws-commands.sh for C2S/SC2S:
- Restore version info comment: "# custom rhcos ami for non-public regions"
- Restore inline comments: "# 4.9 and below" and "# 4.10 and above"
- Add COMPUTE_AMI and echo in C2S block
- Remove unreasonable default COMPUTE_AMI logic outside C2S block
3. Fix multi-nightly.yaml jobs:
a) Rename KMS job to include "etcd" and meet 61-char limit:
aws-eusc-ipi-byo-kms-encryption-fips-tp-amd-f28-destructive
→ aws-eusc-ipi-byo-kms-etcd-encryption-fips-tp-f28-destructive
b) Fix KMS config for destructive job:
ENABLE_AWS_KMS_KEY_COMPUTE/CONTROL_PLANE: yes → no
ENABLE_AWS_KMS_KEY_DEFAULT_MACHINE: no → yes
c) Add -mini-perm to STS job names (they use AWS_INSTALL_USE_MINIMAL_PERMISSIONS):
aws-eusc-ipi-private-sts-tp-arm-f7
→ aws-eusc-ipi-private-sts-mini-perm-tp-arm-f7
aws-eusc-ipi-private-sts-tp-amd-f28-destructive
→ aws-eusc-ipi-private-sts-mini-perm-tp-amd-f28-destructive
Result:
- 8 EUSC jobs in multi-nightly (4 ARM f7 + 4 AMD f28-destructive)
- 4 installer presubmit jobs (unchanged)
- 0 EUSC jobs in amd64-nightly
- Total: 12 EUSC jobs (down from 18)
0de2b19 to
a61aea4
Compare
|
[REHEARSALNOTIFIER]
A total of 16226 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@liweinan: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| @@ -319,31 +319,30 @@ rm /tmp/pull-secret | |||
| # fi | |||
|
|
|||
| # custom rhcos ami for non-public regions | |||
| RHCOS_AMI= | |||
| if [[ "${CLUSTER_TYPE}" =~ ^aws-s?c2s$ ]]; then | |||
| if [[ "${CLUSTER_TYPE}" =~ ^aws-s?c2s$ ]] && [[ -z "${CONTROL_PLANE_AMI}" ]]; then | |||
There was a problem hiding this comment.
| if [[ "${CLUSTER_TYPE}" =~ ^aws-s?c2s$ ]] && [[ -z "${CONTROL_PLANE_AMI}" ]]; then | |
| if [[ "${CLUSTER_TYPE}" =~ ^aws-s?c2s$ ]] && [[ -z "${CONTROL_PLANE_AMI}" ]] && [[ -z "${COMPUTE_AMI}" ]]; then |
| test: | ||
| - chain: openshift-e2e-test-qe | ||
| workflow: baremetal-lab-upi | ||
| - as: aws-eusc-ipi-byo-kms-etcd-encryption-fips-tp-arm-f7 |
There was a problem hiding this comment.
@liweinan this part looks good, please copy these configs to openshift-openshift-tests-private-release-4.23__multi-nightly.yaml and openshift-openshift-tests-private-release-5.0__multi-nightly.yaml as well
|
@tthvo's initial EUSC PR has merged, so we might actually be able to rehearse this, if credentials are setup: /pj-rehearse pull-ci-openshift-installer-main-e2e-aws-eusc-techpreview |
|
@patrickdillon: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Implement continuous integration support for AWS EUSC partition (aws-eusc) in eusc-de-east-1 region. Includes cluster profile definition, service endpoints configuration, custom AMI handling, and periodic test jobs.
This enables OpenShift testing on AWS's new European Sovereign Cloud infrastructure, which requires explicit endpoint configuration and custom RHCOS AMIs not available in public regions.