Litestar Queues
Task queues, workers, schedules, realtime event streams, and storage/execution backends for Litestar.
Quickstart · Architecture · Storage Backends · Execution & Operations · Events & Streams · API Reference
litestar-queues decouples authoritative task storage (BaseQueueBackend) from execution delivery (BaseExecutionBackend). Persist tasks, retries, uniqueness reservations, and event history in SQLSpec (PostgreSQL, MySQL, MariaDB, SQLite, DuckDB, CockroachDB, Spanner, Oracle, SQL Server), Advanced Alchemy, Redis, or Valkey while dispatching execution across local workers, Google Cloud Tasks, Google Cloud Run Jobs, Pub/Sub, SQS, RabbitMQ, or Kafka using an ID-only wire payload ({"version": 1, "task_id": "<uuid>"}).
Install
uv add litestar-queues
Add optional extras for your storage or execution backend (sqlspec, advanced-alchemy, redis, valkey, cloud-tasks, cloudrun, pubsub, sqs, rabbitmq, kafka, otel, prometheus).
Quickstart
from uuid import UUID
from litestar import Litestar, get, post
from litestar.di import NamedDependency
from litestar.exceptions import NotFoundException
from litestar.params import FromPath, FromQuery
from litestar_queues import QueueConfig, QueuePlugin, QueueService, task
@task(queue="reports")
async def render_report(report_id: str, requested_by: str) -> dict[str, str]:
"""Render an analytics report in the background."""
return {"report_id": report_id, "requested_by": requested_by, "status": "ready"}
@post("/reports/{report_id:str}")
async def start_report(
report_id: FromPath[str],
queue_service: NamedDependency[QueueService],
requested_by: FromQuery[str] = "team@example.com",
) -> dict[str, str]:
"""Enqueue a background report job and return its tracking ID."""
handle = await queue_service.enqueue(render_report, report_id, requested_by=requested_by)
return {"task_id": str(handle.id), "status": handle.status or "pending"}
@get("/tasks/{task_id:uuid}")
async def get_report_status(task_id: FromPath[UUID], queue_service: NamedDependency[QueueService]) -> dict[str, object]:
"""Inspect the status and return value of a queued task."""
record = await queue_service.get_task(task_id)
if record is None:
raise NotFoundException(detail=f"Task {task_id} not found")
return {
"task_id": str(record.id),
"status": record.status,
"is_terminal": record.is_terminal,
"result": record.result,
}
app = Litestar(route_handlers=[start_report, get_report_status], plugins=[QueuePlugin(config=QueueConfig())])
Start the server and its supervised worker process:
LITESTAR_APP=app:app uv run litestar run --reload
Enqueue a report job and inspect its status:
curl -X POST http://127.0.0.1:8000/reports/q3-revenue
curl http://127.0.0.1:8000/tasks/<task_id>
Capabilities
| Area | Highlights |
|---|---|
| Storage Backends | Zero-config EphemeralQueueBackend (WAL SQLite), InMemoryQueueBackend, SQLSpecQueueBackend (9+ relational databases & third-party adapters), SQLAlchemyBackend (Advanced Alchemy), RedisQueueBackend, and ValkeyQueueBackend. |
| Execution & Workers | WorkerConfig(placement="server" | "asgi" | "external"), multi-backend routing (QueueConfig.execution_backends), Google Cloud Tasks push callbacks (CloudTasksOIDCGuard), Cloud Run Jobs, Pub/Sub, SQS, RabbitMQ, and Kafka. |
| Tasks & Schedules | Priority queues, retries with exponential/jittered backoff, argument-identity uniqueness & forever reservations, cooperative & remote cancellation, cron/interval ScheduleConfig, and attempt-scoped TaskDependencyProvider. |
| Events & Observability | Live SSE & WebSocket event streams (EventStreamConfig), durable event history (EventHistoryConfig, QueueEventQuery, EventHistoryExtraColumn), and low-cardinality OpenTelemetry, Prometheus, and SQLCommenter telemetry (ObservabilityConfig). |
Documentation
- Getting Started & Installation
- Core Concepts & Guides
- Storage Backends (SQLSpec, Advanced Alchemy, Redis, Valkey)
- Execution Backends, Bounded Maintenance & CLI
- Task Events, Realtime Streams, History & Observability
- Architecture & Runnable Examples
- API & CLI Reference
Contributing & License
Run make install, make test-unit, make lint, and make docs before submitting changes; see the Contributing & Testing guide for details. litestar-queues supports Python 3.11 through 3.15 and is distributed under the MIT License.