Skip to content

Fix MCD image pull failure with mirrored releases#1865

Open
honza wants to merge 1 commit intoopenshift-metal3:masterfrom
honza:mirror-sig-policy
Open

Fix MCD image pull failure with mirrored releases#1865
honza wants to merge 1 commit intoopenshift-metal3:masterfrom
honza:mirror-sig-policy

Conversation

@honza
Copy link
Member

@honza honza commented Mar 11, 2026

RHCOS 9.6 ships a default /etc/containers/policy.json that requires sigstoreSigned verification for quay.io/openshift-release-dev images. When deploying mirrored non-GA releases (nightly/CI), images are not sigstore-signed. Even though registries.conf redirects pulls to the local mirror, the policy check is evaluated against the original image name (quay.io/...), causing signature verification to fail.

This affects two services during node firstboot:

  1. machine-config-daemon-pull.service - the initial MCD image pull
  2. machine-config-daemon-firstboot.service - MCD's internal pulls for extensions and OS images

The existing generate_podman_policy_args.sh only applies a permissive --signature-policy for podman < 4.4.1, but current RHCOS ships podman 5.x. The MCO also internally regenerates the restrictive policy.json during MachineConfig rendering, overriding any file-level MachineConfig entries.

Fix by adding a MachineConfig manifest with:

  • A systemd dropin on machine-config-daemon-pull.service that forces --signature-policy to use the permissive policy-for-old-podman.json
  • A new oneshot service that replaces /etc/containers/policy.json with the permissive version before machine-config-daemon-firstboot runs

Only activated when MIRROR_IMAGES is set and OPENSHIFT_RELEASE_TYPE is not "ga".

RHCOS 9.6 ships a default /etc/containers/policy.json that requires
sigstoreSigned verification for quay.io/openshift-release-dev images.
When deploying mirrored non-GA releases (nightly/CI), images are not
sigstore-signed. Even though registries.conf redirects pulls to the
local mirror, the policy check is evaluated against the original image
name (quay.io/...), causing signature verification to fail.

This affects two services during node firstboot:
1. machine-config-daemon-pull.service - the initial MCD image pull
2. machine-config-daemon-firstboot.service - MCD's internal pulls
   for extensions and OS images

The existing generate_podman_policy_args.sh only applies a permissive
--signature-policy for podman < 4.4.1, but current RHCOS ships
podman 5.x. The MCO also internally regenerates the restrictive
policy.json during MachineConfig rendering, overriding any file-level
MachineConfig entries.

Fix by adding a MachineConfig manifest with:
- A systemd dropin on machine-config-daemon-pull.service that forces
  --signature-policy to use the permissive policy-for-old-podman.json
- A new oneshot service that replaces /etc/containers/policy.json with
  the permissive version before machine-config-daemon-firstboot runs

Only activated when MIRROR_IMAGES is set and OPENSHIFT_RELEASE_TYPE
is not "ga".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci bot requested review from celebdor and cybertron March 11, 2026 18:50
@openshift-ci
Copy link

openshift-ci bot commented Mar 11, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign cybertron for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
Copy link

openshift-ci bot commented Mar 11, 2026

@honza: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-ipi-serial-ovn-ipv6 8064430 link false /test e2e-metal-ipi-serial-ovn-ipv6
ci/prow/e2e-metal-ipi-ovn-ipv6 8064430 link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/e2e-metal-ipi-ovn-dualstack 8064430 link false /test e2e-metal-ipi-ovn-dualstack

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant