Background Tasks
Define, submit, and monitor background tasks with Celery, Redis, and a pluggable notification system for real-time completion alerts.
Singularity ships a complete background task system built on Celery + Redis. Tasks are plain Python functions decorated with @task, auto-discovered at startup, registered as Celery tasks, and executed by a pool of workers. A pluggable notification system dispatches completion events through four built-in channels -- log, callback, websocket, and webhook -- so your services, dashboards, and external integrations stay informed in real time.
Architecture at a Glance
Task Execution Flow
The following diagram shows the full lifecycle of a background task, from service submission through worker execution to notification dispatch.
Notification Channel Dispatch
When a task finishes, every channel listed in the task's notify parameter receives the result. Channels run independently -- one failure does not block the others.
Defining Tasks
The @task Decorator
Every background task is a plain synchronous function wrapped with @task. The decorator registers it in a global registry that TaskRunner discovers at startup.
from singularity.tasks import task
@task(name="send_email", notify=["log"])
def send_email(to: str, subject: str, body: str) -> dict:
"""Send a transactional email."""
# Your email sending logic here
return {"status": "sent", "to": to}Decorator Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
name | str | required | Unique task identifier used for submission and Celery routing |
description | str | "" | Human-readable description (falls back to the function docstring) |
notify | list[str] | [] | Notification channel names: "log", "callback", "websocket", "webhook" |
notify_config | dict | {} | Per-channel configuration. Keys are channel names, values are config objects |
max_retries | int | 3 | Maximum retry attempts on failure |
retry_delay | int | 60 | Seconds between retries |
time_limit | int | 3600 | Hard timeout in seconds (kills the task if exceeded) |
Parameter Introspection
The decorator inspects the function signature at decoration time and stores parameter metadata (name, type, default, required). This powers CLI validation, the tasks info command, and runtime argument checking on submit().
@task(name="process_order", description="Process an e-commerce order")
def process_order(order_id: str, items: list, priority: int = 0) -> dict:
# TaskRunner validates that order_id and items are provided at submit time
return {"order_id": order_id, "total_items": len(items)}Task Organization
Tasks live in tasks/executable/. The runner supports two layouts.
For simple, self-contained tasks -- one file per task.
tasks/executable/
send_email.py # contains @task(name="send_email")
process_order.py # contains @task(name="process_order")For complex tasks that need helper modules -- a directory with __init__.py.
tasks/executable/
generate_report/
__init__.py # may contain @task(name="generate_report")
pdf_builder.py # helper module
chart_renderer.py # helper moduleAll .py files inside the directory are imported, so the @task decorator can live in any file within the package.
Notification Channels
Singularity includes four built-in channels. You can also register custom ones.
Emits a structured Loguru message with task name, status, duration, result, and safe meta fields.
@task(name="cleanup", notify=["log"])
def cleanup(days: int = 30) -> dict:
deleted = purge_old_records(days)
return {"deleted": deleted}Output on success:
INFO | Task completed | task_name=cleanup status=success duration_ms=142.57 result={'deleted': 83}Output on failure:
ERROR | Task failed | task_name=cleanup status=failure duration_ms=23.10 error="ConnectionRefusedError..."Invokes a Python function registered via @task_fn.on_complete. The callback runs synchronously inside the Celery worker after the task finishes. Two callback sources are checked (both are called if present):
- The decorator-registered
on_completefunction - A callable passed at submit time via
_on_complete
@task(name="generate_report", notify=["callback"])
def generate_report(report_type: str, date_range: dict) -> dict:
return {"file": f"/reports/{report_type}.pdf"}
@generate_report.on_complete
def on_report_done(status: str, result: dict, meta: dict):
if status == "success":
# Chain: send the report via email
send_email.submit(
to=meta.get("requester_email", "admin@example.com"),
subject=f"Report ready: {result['file']}",
body="Your report has been generated.",
)Publishes a JSON message to a Redis Pub/Sub channel named task_notifications. A listener inside the FastAPI process subscribes and broadcasts through the WebSocketManager, bridging the Celery worker process and the FastAPI application that owns the live WebSocket connections.
@task(name="long_import", notify=["websocket"])
def long_import(file_path: str) -> dict:
rows = process_csv(file_path)
return {"imported": rows}The published message includes routing fields from meta:
{
"type": "task_completed",
"task_name": "long_import",
"status": "success",
"result": {"imported": 1000},
"duration_ms": 5432.10,
"org_id": "org_abc",
"user_id": "usr_123"
}Pass org_id and user_id in _meta at submit time to enable targeted delivery to specific WebSocket connections.
Sends an HTTP request (POST or PUT) to an external URL with the task result as a JSON payload. Configuration is provided via the WebhookConfig dataclass.
from singularity.tasks.notifications.webhook_channel import WebhookConfig
slack_config = WebhookConfig(
url="https://hooks.slack.com/services/T00/B00/xxxxx",
headers={"Content-Type": "application/json"},
)
@task(
name="process_order",
notify=["log", "webhook"],
notify_config={"webhook": slack_config},
max_retries=5,
retry_delay=120,
)
def process_order(order_id: str, items: list) -> dict:
total = sum(i["price"] for i in items)
return {"order_id": order_id, "total": total}WebhookConfig fields:
| Field | Type | Default | Description |
|---|---|---|---|
url | str | required | The webhook endpoint URL |
headers | dict[str, str] | {} | HTTP headers (Authorization, API keys, etc.) |
method | str | "POST" | HTTP method -- "POST" or "PUT" |
timeout | int | 10 | Request timeout in seconds |
You can also pass a simple webhook_url in _meta at submit time as a lightweight alternative to WebhookConfig. The channel falls back to this if no notify_config is set.
Submitting Tasks from Services
The TaskRunner is available in every service via acquire.tasks. Call .submit() with the task name and keyword arguments.
from services.__base.service import Service
class ReportsService(Service):
def __init__(self, acquire):
super().__init__(acquire)
self.tasks = acquire.tasks
async def post(self, report_type: str):
task_ref = self.tasks.submit(
"generate_report",
report_type=report_type,
date_range={"start": "2024-01-01", "end": "2024-12-31"},
_meta={"requester_email": "user@example.com"},
)
return {"task_id": task_ref.id, "status": "queued"}submit() Parameters
| Parameter | Type | Description |
|---|---|---|
task_name | str | The unique task identifier (must match @task(name=...)) |
*args | positional | Forwarded to the task function |
**kwargs | keyword | Forwarded to the task function |
_meta | dict | Contextual metadata passed to notification channels (not forwarded to the task) |
_countdown | int | Delay in seconds before the task starts executing |
_on_complete | callable | Ad-hoc callback invoked on completion (in addition to any @task.on_complete) |
submit() validates that all required parameters are satisfied before sending the task to the broker. If a required parameter is missing, it raises a TypeError with a clear description of the expected signature.
Completion Callbacks and Chaining
The callback system enables task chaining without coupling tasks to each other. Register a callback with the @task_fn.on_complete decorator:
@task(name="generate_invoice", notify=["callback", "log"])
def generate_invoice(order_id: str) -> dict:
pdf_path = create_invoice_pdf(order_id)
return {"invoice_path": pdf_path, "order_id": order_id}
@generate_invoice.on_complete
def after_invoice(status: str, result: dict, meta: dict):
"""Chain: email the invoice, then notify via Slack."""
if status == "success":
# Submit another task -- creates a chain
send_email.submit(
to=meta.get("customer_email"),
subject=f"Invoice for order {result['order_id']}",
body=f"Your invoice is ready: {result['invoice_path']}",
)The callback receives three arguments:
| Argument | Type | Description |
|---|---|---|
status | str | "success" or "failure" |
result | dict | The task's return value (on success) or None (on failure) |
meta | dict | The _meta dict from submit(), plus internal keys like _on_complete_fn |
Custom Notification Channels
Extend the notification system by implementing the NotificationChannel abstract base class and registering it with the TaskRunner.
from singularity.tasks.notifications import NotificationChannel
class PagerDutyChannel(NotificationChannel):
"""Alert PagerDuty on task failure."""
def send(
self,
task_name: str,
status: str,
result=None,
error=None,
duration_ms: float = 0,
meta=None,
task_config=None,
) -> None:
if status != "failure":
return
# Send PagerDuty alert
requests.post(
"https://events.pagerduty.com/v2/enqueue",
json={
"routing_key": task_config.get("routing_key"),
"event_action": "trigger",
"payload": {
"summary": f"Task {task_name} failed: {error}",
"severity": "critical",
"source": "singularity-tasks",
},
},
)Register it during application startup:
from singularity.tasks.runner import TaskRunner
runner = TaskRunner()
runner.register_notification_channel("pagerduty", PagerDutyChannel())Then use it in any task:
@task(
name="critical_sync",
notify=["log", "pagerduty"],
notify_config={"pagerduty": {"routing_key": "R0123456789"}},
)
def critical_sync(source: str) -> dict:
...Retry and Error Handling
Tasks automatically retry on failure up to max_retries times, with retry_delay seconds between each attempt. The Celery worker handles retry scheduling.
@task(
name="call_external_api",
max_retries=5,
retry_delay=120, # 2 minutes between retries
time_limit=300, # kill after 5 minutes
notify=["log", "webhook"],
notify_config={"webhook": alert_config},
)
def call_external_api(endpoint: str, payload: dict) -> dict:
response = requests.post(endpoint, json=payload, timeout=30)
response.raise_for_status()
return response.json()The time_limit is a hard kill. If your task is interrupted by a time limit, the worker terminates the process and the task status becomes failure. Notifications fire in the finally block, so even timed-out tasks dispatch their configured channels.
Worker Management
-
Start a worker with the default thread pool:
tasks worker -
Increase concurrency for CPU-bound workloads:
tasks worker --concurrency 8 -
Switch pool type for I/O-bound tasks:
tasks worker --pool gevent -
Enable the Beat scheduler for periodic tasks:
tasks worker --beat -
Use autoscaling to dynamically adjust the worker count:
tasks worker --autoscale 10,2
Worker CLI Options
| Option | Type | Default | Description |
|---|---|---|---|
--concurrency | int | 2 | Number of concurrent worker processes/threads |
--loglevel | choice | info | Logging level: debug, info, warning, error, critical |
--queues | str | celery | Comma-separated list of queues to consume from |
--hostname | str | auto | Custom hostname for the worker |
--max-tasks-per-child | int | 1000 | Max tasks before a child process is replaced |
--prefetch-multiplier | int | 1 | Number of messages to prefetch per worker |
--pool | choice | threads | Pool implementation: prefork, eventlet, gevent, threads, solo |
--autoscale | str | -- | Autoscaling range as max,min |
--beat | flag | -- | Enable the Celery Beat scheduler alongside the worker |
Task CLI Commands
The tasks CLI provides commands for listing, inspecting, creating, running, and checking the status of background tasks.
# List all discovered tasks
tasks list
# Detailed info about a specific task (shows parameters, types, defaults)
tasks info send_email
# Create a new task with boilerplate
singularity generate task send_notification -d "Send push notifications to users"
# Trigger a task manually with keyword arguments
tasks run send_email -k '{"to": "user@example.com", "subject": "Hello", "body": "World"}'
# Trigger with a countdown delay (starts in 60 seconds)
tasks run process_order -k '{"order_id": "ord_123", "items": []}' -c 60
# Check the result of a submitted task
tasks result abc12345-task-idComplete Example
Putting it all together -- a task with webhook notifications, a completion callback, and service-layer submission:
# tasks/executable/process_payment.py
from singularity.tasks import task
from singularity.tasks.notifications.webhook_channel import WebhookConfig
slack_webhook = WebhookConfig(
url="https://hooks.slack.com/services/T00/B00/xxxxx",
headers={"Content-Type": "application/json"},
)
@task(
name="process_payment",
description="Charge a customer and update their subscription",
notify=["log", "callback", "webhook"],
notify_config={"webhook": slack_webhook},
max_retries=3,
retry_delay=60,
time_limit=120,
)
def process_payment(user_id: str, amount: float, currency: str = "usd") -> dict:
"""Charge the customer via Stripe and activate their subscription."""
charge = stripe.Charge.create(
amount=int(amount * 100),
currency=currency,
customer=user_id,
)
return {"charge_id": charge.id, "amount": amount, "user_id": user_id}
@process_payment.on_complete
def on_payment_done(status: str, result: dict, meta: dict):
if status == "success":
# Chain: send a receipt email
from tasks.executable.send_email import send_email
send_email.submit(
to=meta.get("customer_email"),
subject="Payment received",
body=f"We charged ${result['amount']} to your card.",
)# services/v1/billing/service.py
from services.__base.service import Service
class BillingService(Service):
def __init__(self, acquire):
super().__init__(acquire)
self.tasks = acquire.tasks
async def post(self, user_id: str, amount: float):
ref = self.tasks.submit(
"process_payment",
user_id=user_id,
amount=amount,
_meta={
"customer_email": "customer@example.com",
"org_id": "org_abc",
},
)
return {"task_id": ref.id, "status": "processing"}