[Bug] - Missing Packages for amazon-linux-repo-cdn-2023 and ncurses-libs

[phene]

Describe the bug
It appears the package index was updated but packages have not been made available yet for new versions of amazon-linux-repo-cdn-2023 and ncurses-libs

To Reproduce

1. Download latest AmazonLinux 2023 container image
2. Perform dnf upgrade -y --releasever latest
3. See failures

1.912 Amazon Linux 2023 repository                     35 MB/s |  56 MB     00:01
9.549 Last metadata expiration check: 0:00:08 ago on Wed Mar 25 19:48:15 2026.
10.12 Dependencies resolved.
10.14 ================================================================================
10.14  Package               Arch    Version                        Repository   Size
10.14 ================================================================================
10.14 Upgrading:
10.14  amazon-linux-repo-cdn noarch  2023.10.20260316-0.amzn2023    amazonlinux  15 k
10.14  ncurses-base          noarch  6.6-1.amzn2023.0.1             amazonlinux  65 k
10.14  ncurses-libs          aarch64 6.6-1.amzn2023.0.1             amazonlinux 331 k
10.14  system-release        noarch  2023.10.20260316-0.amzn2023    amazonlinux  27 k
10.14
10.14 Transaction Summary
10.14 ================================================================================
10.14 Upgrade  4 Packages
10.14
10.14 Total download size: 439 k
10.14 Downloading Packages:
10.31 [MIRROR] amazon-linux-repo-cdn-2023.10.20260316-0.amzn2023.noarch.rpm: Status code: 403 for https://cdn.amazonlinux.com/blobstore/74bff79766e897ddb78d6bc3fb2c7f86c336a3899227deeef4bf310c0431317a/amazon-linux-repo-cdn-2023.10.20260316-0.amzn2023.noarch.rpm?instance_id=none (IP: 65.8.54.55)
10.34 [MIRROR] amazon-linux-repo-cdn-2023.10.20260316-0.amzn2023.noarch.rpm: Status code: 403 for https://cdn.amazonlinux.com/blobstore/74bff79766e897ddb78d6bc3fb2c7f86c336a3899227deeef4bf310c0431317a/amazon-linux-repo-cdn-2023.10.20260316-0.amzn2023.noarch.rpm?instance_id=none (IP: 65.8.54.55)
10.36 [MIRROR] ncurses-libs-6.6-1.amzn2023.0.1.aarch64.rpm: Status code: 403 for https://cdn.amazonlinux.com/blobstore/fa0fe20a992ea91574491c13a46dd87134a61ae51c8ed6b27fad756c89222c10/ncurses-libs-6.6-1.amzn2023.0.1.aarch64.rpm?instance_id=none (IP: 65.8.54.55)
10.40 [MIRROR] ncurses-libs-6.6-1.amzn2023.0.1.aarch64.rpm: Status code: 403 for https://cdn.amazonlinux.com/blobstore/fa0fe20a992ea91574491c13a46dd87134a61ae51c8ed6b27fad756c89222c10/ncurses-libs-6.6-1.amzn2023.0.1.aarch64.rpm?instance_id=none (IP: 65.8.54.55)
10.40 [FAILED] ncurses-libs-6.6-1.amzn2023.0.1.aarch64.rpm: No more mirrors to try - All mirrors were already tried without success
10.40
10.40 The downloaded packages were saved in cache until the next successful transaction.
10.40 You can remove cached packages by executing 'dnf clean packages'.
10.41 Error: Error downloading packages:
10.41   ncurses-libs-6.6-1.amzn2023.0.1.aarch64: Cannot download, all mirrors were already tried without success

Expected behavior
I expected the packages to update successfully

[esousa-zup]

I'm getting the same error.

All environment that have this configuration to ensure the last update are breaking; Until fix, I removed the dnf update -y --releasever=latest

[alexey-tsvetnov]

We're looking into this

[sktomer]

We have identified the root cause and are working on a fix.

[phene]

Any ETA on a fix?

[sktomer]

This should be fixed now. Please report if you still see issues.

[phene]

The issue appears to be resolved. Thanks!

[alexey-tsvetnov]

FYI, https://docs.aws.amazon.com/linux/al2023/ug/updating-best-practice.html

Warning
Running "dnf --releasever=latest update" is not best practice,
and is likely to result in an OS update being first tested in production.

Instead of using latest, use a specific AL2023 release version.
This ensures you are deploying the same changes
across production instances as you previously tested.
For example, "dnf --releasever=2023.10.20260216 update"
will always update to the 2023.10.20260216 release.

[phene]

I'm opposed to in-place updates in production at all. Our build pipelines have been unhappy much of today, though.

[ashishnegi]

@alexey-tsvetnov Thanks for the quick turnaround.
What was the root cause ? Has it been fixed ?

[chadmyers]

FYI, https://docs.aws.amazon.com/linux/al2023/ug/updating-best-practice.html

Warning
Running "dnf --releasever=latest update" is not best practice,
and is likely to result in an OS update being first tested in production.

Instead of using latest, use a specific AL2023 release version.
This ensures you are deploying the same changes
across production instances as you previously tested.
For example, "dnf --releasever=2023.10.20260216 update"
will always update to the 2023.10.20260216 release.

How do people stay on top of CVE patching SLAs if the patched version of packages aren't available in that specific release? The only way I've found to be able to patch CVEs in a timely manner is to always use --releasever latest.

[awsthk]

FYI, https://docs.aws.amazon.com/linux/al2023/ug/updating-best-practice.html

Warning
Running "dnf --releasever=latest update" is not best practice,
and is likely to result in an OS update being first tested in production.

Instead of using latest, use a specific AL2023 release version.
This ensures you are deploying the same changes
across production instances as you previously tested.
For example, "dnf --releasever=2023.10.20260216 update"
will always update to the 2023.10.20260216 release.

How do people stay on top of CVE patching SLAs if the patched version of packages aren't available in that specific release? The only way I've found to be able to patch CVEs in a timely manner is to always use --releasever latest.

It is usually better to update using --releasever=latest in a dedicated test environment first. When you don't see any errors in your test environment doing that, you would update your production environment using --releasever=<the_release_version_you_tested>. That way, it is less likely that a broken update brings down your production systems, and if you stretch out your testing a bit longer, you won't unexpectedly deploy an untested new release to your production fleet.

[alexey-tsvetnov]

Thanks for the quick turnaround.

Thank you all for a quick report!

What was the root cause ? Has it been fixed ?

The issue with the latest deployment was identified and quickly rolled back to a previous state to mitigate it.

At this particular moment it is 2023.10.20260302, but we are working on getting a new AL2023 release back ASAP:

# cat /etc/amazon-linux-release
Amazon Linux release 2023.10.20260302 (Amazon Linux)

# dnf upgrade -y --releasever latest
Dependencies resolved.
Nothing to do.
Complete!

[ashishnegi]

How do people stay on top of CVE patching SLAs if the patched version of packages aren't available in that specific release? The only way I've found to be able to patch CVEs in a timely manner is to always use --releasever latest.

One way is to upload the image after yum update to a container registry. Then configure your environments to pull the image from the registry. This way, you are building with latest packages but doing a phased rollout across dev/stage environments before rolling it out in production.