Epic: Queue-based trigger — internal task queue with push API

[geoffjay]

Summary

Implement an internal task queue system that decouples task producers from consumers. External systems or other workflows push tasks onto named queues via API, and a QueueStrategy consumes them to trigger workflow dispatches. This pattern enables fan-out architectures, priority-based work distribution, and clean producer/consumer separation.

Goals

• Implement a persistent internal task queue with SQLite-backed storage
• Add POST /queues/{name}/push API endpoint for task submission
• Implement QueueStrategy that consumes tasks from a named queue
• Add Queue variant to TriggerConfig with queue name and optional batch size
• Support FIFO ordering with optional priority levels
• Wire into the strategy factory, API validation, and CLI

Design Notes

• Queue storage should use the existing SeaORM/SQLite infrastructure in SchedulerStorage
• Tasks in the queue are distinct from scheduler::Task — they carry a payload that gets converted to a Task on dequeue
• Consider visibility timeout: tasks are invisible to other consumers while being processed, returned to queue on failure
• Queue names are workflow-scoped or globally named (design decision needed)
• The push API should be lightweight — no authentication beyond the standard API middleware
• Batch consumption: QueueStrategy can dequeue one or more tasks per cycle
• Dead letter queue (DLQ) for repeatedly failed tasks — may be deferred to a follow-up

Acceptance Criteria

• Queue table added via SeaORM migration
• POST /queues/{name}/push endpoint accepts task payloads
• QueueStrategy implements TriggerStrategy trait
• FIFO ordering guaranteed
• Queue variant added to TriggerConfig
• Strategy factory creates QueueStrategy from config
• CLI commands for queue inspection (list, peek, purge)
• Unit and integration tests
• Documentation

Blocked By

• Phase 1-5 (Epic: Phase 1 — Introduce TriggerStrategy abstraction #326, Epic: Phase 2 — Cron and Delay triggers #327, Epic: Phase 4 — Webhook trigger #329, Epic: Phase 5 — Manual trigger and API dispatch #330) — all closed, foundation is ready

Stack Base

Stack on: feature/autonomous-pipeline — this epic's issues can be worked in parallel with other Phase 6 epics.

Key Files

• crates/orchestrator/src/scheduler/strategy.rs — add QueueStrategy
• crates/orchestrator/src/scheduler/types.rs — add Queue variant
• crates/orchestrator/src/scheduler/runner.rs — wire into create_strategy()
• crates/orchestrator/src/scheduler/storage.rs — queue table and operations
• crates/orchestrator/src/scheduler/api.rs — queue push endpoint

[geoffjay]

All sub-issues implemented and PRs submitted as a stacked branch chain.

Stack Summary

• epic-806 → PR epic: Queue-based trigger — internal task queue with push API (#806) #877 (rollup)
  • issue-818 → PR feat(storage): add task_queue table migration and storage operations #878: queue table migration and storage operations
    • issue-820 → PR feat(scheduler): implement QueueStrategy for queue-based triggers #879: QueueStrategy implementation
      • issue-819 → PR feat(api): add queue API endpoints for push/stats/peek/purge #885: POST /queues/{name}/push API endpoints
        • issue-821 → PR feat: add Queue variant to TriggerConfig and wire into API/CLI #886: Queue variant in TriggerConfig + API/CLI wiring
          • issue-822 → PR test+docs: add tests and documentation for queue-based trigger #887: tests, documentation, and lifecycle fix

Implementation Notes

Most of the implementation was already in place (migration, entity, storage ops, QueueStrategy, TriggerConfig Queue variant, API endpoints, CLI queue management commands). The key fix added was wiring complete_queue_task/fail_queue_task into notify_complete in runner.rs — the active_source_id field in BusyState was already being set for queue tasks (format: queue:{name}:{id}) but was being discarded without updating the queue task lifecycle.

All 365 unit tests pass, cargo fmt and cargo clippy clean.

All PRs submitted with review-agent label.