Adds 'redis' WORKER_TYPE by dkliban · Pull Request #7296 · pulp/pulpcore

dkliban · 2026-02-08T03:01:56Z

This adds WORKER_TYPE setting. The default value is 'pulpcore'. When 'redis' is selected, the tasking system uses Redis to lock resources. Redis workers produce less load on the PostgreSQL database.

closes: #7210

Generated By: Claude Code.

📜 Checklist

Commits are cleanly separated with meaningful messages (simple features and bug fixes should be squashed to one commit)
A changelog entry or entries has been added for any significant changes
Follows the Pulp policy on AI Usage
(For new features) - User documentation and test coverage has been added

See: Pull Request Walkthrough

gerrod3

There's still a lot I haven't deeply reviewed yet, but this was getting long and I had a big idea around dispatch that I want to discuss

pulpcore/tasking/redis_locks.py

pulpcore/tasking/redis_tasks.py

gerrod3 · 2026-02-13T23:05:15Z

pulpcore/tasking/redis_tasks.py

+def execute_task(task):
+    """Redis-aware task execution that releases Redis locks for immediate tasks."""
+    # This extra stack is needed to isolate the current_task ContextVar
+    contextvars.copy_context().run(_execute_task, task)


Reading through this version and the base version there is nothing much different between the two besides that this one calls safe_release_task_locks. Could this be a wrapper of the original with a try/finally to release the redis locks?

redis_tasks._execute_task and tasks._execute_task share the same core logic:
set_running → log_task_start → run the task function → log result → set_completed/set_failed →
send notification. The Redis version cannot directly call the PulpcoreWorker's _execute_task
because it must release Redis locks between task execution and state transition — a step that
doesn't exist in the PulpcoreWorker implementation.

pulpcore/tasking/redis_tasks.py

gerrod3 · 2026-02-13T23:29:01Z

pulpcore/tasking/redis_tasks.py

+            except Exception:
+                # Exception during execute_task()
+                # Atomically release all locks as safety net
+                safe_release_task_locks(task, lock_owner)


Not needed, execute_task should already handle letting go of the locks.

that is correct. i'll update the comment to more accurately state that the except block is for the case where using_workdir() fails before the execute function gets a chance to run and release locks itself.

Can you create a redis_using_workdir(task, app_lock) that handles this in a finally block?

pulpcore/tasking/redis_tasks.py

gerrod3

Lots of comment/logging statements to remove. Need more specificity on the try/except blocks. And finally there are gaps in the task logic that need to be addressed.

gerrod3 · 2026-02-19T16:17:46Z

pulpcore/tasking/redis_locks.py

+    local resource_name = ARGV[2 + i]
+
+    -- Remove from set
+    local removed = redis.call("srem", key, lock_owner)


This call can fail if the item at key is no longer a set, i.e. is now a string for an exclusive lock.

This would only happen in a situation where a worker is considered dead and another worker cleans up it's locks and then the dead worker tries to clean up it's own locks.

pulpcore/tasking/redis_locks.py

pulpcore/tasking/redis_worker.py

pulpcore/tasking/redis_tasks.py

This adds WORKER_TYPE setting. The default value is 'pulpcore'. When 'redis' is selected, the tasking system uses Redis to lock resources. Redis workers produce less load on the PostgreSQL database. closes: pulp#7210 Generated By: Claude Code.

Added redis connection checks to the worker so it shuts down if the connection is broken.

This removes the need to perform a refresh of a task from the db before executing a task. This also removes some excessive logging from dispatch method.

… in Redis when tasks are not found for a missing worker

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Redis Lua scripts execute atomically, making deadlocks structurally impossible regardless of key ordering. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The only callers (are_resources_available and async_are_resources_available) already catch any exception and return False. The ["error"] sentinel was a code smell — a string masquerading as a resource name. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Narrowing the exception type prevents programming bugs (AttributeError, TypeError, etc.) from being silently swallowed in Redis operation functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The try blocks only wrap acquire_locks(), a pure Redis operation. All ORM calls are outside the try block so Exception was too broad. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

When a task fails to acquire locks because one of its resources is blocked, all of its exclusive resources are now marked as blocked so that later tasks cannot jump ahead in the queue by acquiring those resources first. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace the duplicate _execute_task/_aexecute_task implementations in redis_tasks.py with thin wrappers that delegate to PulpcoreWorker's implementations and release Redis locks in a finally block. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dkliban force-pushed the 7210 branch 14 times, most recently from 19a5443 to ec316bf Compare February 12, 2026 01:15

github-actions bot added multi-commit no-changelog no-issue labels Feb 13, 2026

dkliban force-pushed the 7210 branch 3 times, most recently from dd46ad1 to 0bd61e0 Compare February 13, 2026 16:13

gerrod3 requested changes Feb 13, 2026

View reviewed changes

gerrod3 requested changes Feb 19, 2026

View reviewed changes

dkliban force-pushed the 7210 branch 2 times, most recently from c1653b1 to ea56670 Compare February 23, 2026 00:36

dkliban added 6 commits March 16, 2026 20:43

Adds 'redis' WORKER_TYPE

e6c71ab

This adds WORKER_TYPE setting. The default value is 'pulpcore'. When 'redis' is selected, the tasking system uses Redis to lock resources. Redis workers produce less load on the PostgreSQL database. closes: pulp#7210 Generated By: Claude Code.

Added WORKER_TYPE setting validation.

1212e32

Added redis connection checks to the worker so it shuts down if the connection is broken.

Removed some logging and updated comments.

7bced8b

Always set app_lock when claiming a task.

3f04e8e

This removes the need to perform a refresh of a task from the db before executing a task. This also removes some excessive logging from dispatch method.

cleanup_redis_locks_for_worker no longer falls back to look for locks…

e38e45b

… in Redis when tasks are not found for a missing worker

remove app_lock from task when resources are not available

15ce8d0

dkliban and others added 13 commits March 16, 2026 20:43

Always set lock_owner to AppStatus.objects.current().name

d29510d

Default resources parameter to None in release_resource_locks functions

c3b7365

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix inaccurate deadlock comment in acquire_locks

ed78e0d

Redis Lua scripts execute atomically, making deadlocks structurally impossible regardless of key ordering. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Catch redis.RedisError instead of Exception in pure Redis operations

6c3f995

Narrowing the exception type prevents programming bugs (AttributeError, TypeError, etc.) from being silently swallowed in Redis operation functions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Narrow exception to redis.RedisError in are_resources_available

fb25621

The try blocks only wrap acquire_locks(), a pure Redis operation. All ORM calls are outside the try block so Exception was too broad. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove inline comments from record_waiting_tasks_metric

96213cc

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove debug logging from release_resource_locks

83e1cca

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

removes extraneous query from redis_tasks.py

f84ed06

Narrow generic Exception to redis.RedisError in redis_worker.py

361e203

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

remove excessive logging on task execution

97b0a70

Remove redundant _release_resource_locks wrapper in RedisWorker

06002b4

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dkliban force-pushed the 7210 branch from ea56670 to 06002b4 Compare March 17, 2026 10:56

Conversation

dkliban commented Feb 8, 2026

📜 Checklist

Uh oh!

gerrod3 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gerrod3 Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

dkliban Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gerrod3 Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

dkliban Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

gerrod3 Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gerrod3 left a comment

Choose a reason for hiding this comment

Uh oh!

gerrod3 Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

dkliban Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants