Skip to content

fix(queue): allow QUEUE_WORKERS env var to override hardcoded worker count#429

Open
scnerd wants to merge 5 commits intorishikanthc:mainfrom
scnerd:fix/queue-workers-env
Open

fix(queue): allow QUEUE_WORKERS env var to override hardcoded worker count#429
scnerd wants to merge 5 commits intorishikanthc:mainfrom
scnerd:fix/queue-workers-env

Conversation

@scnerd
Copy link
Copy Markdown

@scnerd scnerd commented Mar 14, 2026

Summary

The QUEUE_WORKERS env var is supposed to control how many queue workers get created. This is handled by internal/queue/queue.go:getOptimalWorkerCount which checks:

	// Check for environment variable override
	if workerStr := os.Getenv("QUEUE_WORKERS"); workerStr != "" {
		if workers, err := strconv.Atoi(workerStr); err == nil && workers > 0 {
			return workers, workers // Fixed worker count
		}
	}

However, when this value gets used, it's overwritten by a legacy workers argument without checking if the env var was set in internal/queue/queue.go:NewTaskQueue:

	min, max := getOptimalWorkerCount()
	if legacyWorkers > 0 {
		min = legacyWorkers
		max = legacyWorkers
	}

Since NewTaskQueue is called unconditionally with legacyWorkers = 2 (see main.go:130):

	taskQueue := queue.NewTaskQueue(2, unifiedProcessor, jobRepo) // 2 workers

this means that QUEUE_WORKERS is always ignored.

The right solution would seem to be to remove the legacyWorkers argument entirely, since the behavior of selecting worker pool size ranges and auto-scaling is now handled in getOptimalWorkerCount() instead of in main.go. However, the fact that this seemingly-simple refactor wasn't done when getOptimalWorkerCount was first added suggests that it's not as simple as it seems, and as a new contributor, I'll err on the side of caution. As such, this PR only proposes ignoring legacyWorkers when QUEUE_WORKERS is explicitly set. I've left a TODO comment recommending that legacyWorkers be properly deprecated and removed when practical.

This development followed TDD. Tests were verified to fail before changes and work after them.

Fixes #379

Fixes #417

Fixes #428

Disclaimer: Used Claude Code to find bug and implement changes. All code and comments are fully manually reviewed.

scnerd and others added 5 commits March 13, 2026 22:39
Add tests verifying that getOptimalWorkerCount() respects the
QUEUE_WORKERS environment variable and that NewTaskQueue() should
allow QUEUE_WORKERS to override the hardcoded legacy worker count.

Includes a failing test (TestNewTaskQueue_EnvOverridesLegacy) that
reproduces the bug where QUEUE_WORKERS is always overridden by the
hardcoded legacyWorkers parameter.

Ref: rishikanthc#379

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…count

The QUEUE_WORKERS environment variable was defined and read in
getOptimalWorkerCount(), but NewTaskQueue() unconditionally overwrote
the result with the hardcoded legacyWorkers parameter (always 2).
This made QUEUE_WORKERS effectively dead code.

Now legacyWorkers is only used as a fallback when QUEUE_WORKERS is
not set, preserving the default of 2 workers while allowing users
to control concurrency via the environment variable.

Set QUEUE_WORKERS=1 to serialize all transcription jobs and prevent
system overload during bulk uploads.

Fixes: rishikanthc#379

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch from raw t.Errorf to testify/assert for consistency with the
rest of the codebase. Use t.Setenv() instead of manual os.Setenv/defer
os.Unsetenv for automatic cleanup. Simplify table structs where min
and max are always equal.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All six test functions now have // TestFoo verifies... comments
matching the project's existing convention.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…debt and we should just remove the legacy workers argument entirely.
@scnerd scnerd marked this pull request as ready for review March 14, 2026 13:18
@scnerd scnerd marked this pull request as draft March 15, 2026 02:41
@Iconic06
Copy link
Copy Markdown

This would also fix #417 and #428

@scnerd
Copy link
Copy Markdown
Author

scnerd commented Mar 17, 2026

Stuck as WIP while I work through manual testing. This doesn't seem to have resolved the issue in my local setup yet.

@scnerd
Copy link
Copy Markdown
Author

scnerd commented Mar 20, 2026

Issue was with my testing (was forgetting to build docker image from the correct source), not with the code.

Demo of QUEUE_WORKERS=1:

image

@scnerd scnerd marked this pull request as ready for review March 20, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants