Cross-replica atomicity and coordination

[dralley]

Is your feature request related to a problem? Please describe.

Various replicas of an Pulp instance can update out-of-sync with each other.

Describe the solution you'd like

As a "next step" to #7333, which enabled intra-replica atomicity, it would be nice if distribution updates could be co-ordinated across replicas such that all replicas would update their distributed content sets simultaneously and atomically.

There is also a potential use case for the final distribution update to optionally be separate and manually triggered - as this would allow the external management application to trigger replication on a number of replicas and once all replicas report that they are ready to update their distributions, that external application can trigger that distribution update on all replicas simultaneously.

Challenges: coordination of large systems is always difficult

• how are failures handled and propagated?
• does a failed or stuck replica become a problem for the entire replica ecosystem?
• should this be optional, should there be an override for emergency situations where one might not want to wait or be denied access to updated content over coordination issues?

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context about the feature request here.