[Feature]: partitions, virtual queues by some key

[kirillsalykin]

Is your feature request related to a problem?

All changes related to one entity are ordered and should be executed in the same order to avoid race conditions

I am looking for something similar to Kafka’s key, meaning the ability to ensure ordering within a partition.

It would be great if Apalis allowed only one active job for a particular key.

For example, all jobs related to order 123 should be executed sequentially, which would guarantee that jobs do not overwrite each other’s results.

Describe the solution you'd like

WorkerBuilder::new("emails")
    .backend(storage)
    .parition_by(|job: &SendEmail| job.account_id)

Backend picks next job taking partition key into account + skips parition keys which currently are locked.

Describe alternatives you've considered

No response

Use case

As I mentioned before, all changes related to one entity are ordered and should be executed in the same order to avoid race conditions

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[geofmureithi]

This is possibly just relevant to Postgres and Redis?

[kirillsalykin]

yes, probably mysql as well

so indeed backend specific thing

inmemory backend not affected by this probably because the order of execution is sequential