Singularity
Guides

Background Tasks

Define, submit, and monitor background tasks with Celery, Redis, and a pluggable notification system for real-time completion alerts.

Singularity ships a complete background task system built on Celery + Redis. Tasks are plain Python functions decorated with @task, auto-discovered at startup, registered as Celery tasks, and executed by a pool of workers. A pluggable notification system dispatches completion events through four built-in channels -- log, callback, websocket, and webhook -- so your services, dashboards, and external integrations stay informed in real time.


Architecture at a Glance

Task Execution Flow

The following diagram shows the full lifecycle of a background task, from service submission through worker execution to notification dispatch.

Notification Channel Dispatch

When a task finishes, every channel listed in the task's notify parameter receives the result. Channels run independently -- one failure does not block the others.


Defining Tasks

The @task Decorator

Every background task is a plain synchronous function wrapped with @task. The decorator registers it in a global registry that TaskRunner discovers at startup.

from singularity.tasks import task

@task(name="send_email", notify=["log"])
def send_email(to: str, subject: str, body: str) -> dict:
    """Send a transactional email."""
    # Your email sending logic here
    return {"status": "sent", "to": to}

Decorator Parameters

ParameterTypeDefaultDescription
namestrrequiredUnique task identifier used for submission and Celery routing
descriptionstr""Human-readable description (falls back to the function docstring)
notifylist[str][]Notification channel names: "log", "callback", "websocket", "webhook"
notify_configdict{}Per-channel configuration. Keys are channel names, values are config objects
max_retriesint3Maximum retry attempts on failure
retry_delayint60Seconds between retries
time_limitint3600Hard timeout in seconds (kills the task if exceeded)

Parameter Introspection

The decorator inspects the function signature at decoration time and stores parameter metadata (name, type, default, required). This powers CLI validation, the tasks info command, and runtime argument checking on submit().

@task(name="process_order", description="Process an e-commerce order")
def process_order(order_id: str, items: list, priority: int = 0) -> dict:
    # TaskRunner validates that order_id and items are provided at submit time
    return {"order_id": order_id, "total_items": len(items)}

Task Organization

Tasks live in tasks/executable/. The runner supports two layouts.

For simple, self-contained tasks -- one file per task.

tasks/executable/
  send_email.py        # contains @task(name="send_email")
  process_order.py     # contains @task(name="process_order")

For complex tasks that need helper modules -- a directory with __init__.py.

tasks/executable/
  generate_report/
    __init__.py        # may contain @task(name="generate_report")
    pdf_builder.py     # helper module
    chart_renderer.py  # helper module

All .py files inside the directory are imported, so the @task decorator can live in any file within the package.


Notification Channels

Singularity includes four built-in channels. You can also register custom ones.

Emits a structured Loguru message with task name, status, duration, result, and safe meta fields.

@task(name="cleanup", notify=["log"])
def cleanup(days: int = 30) -> dict:
    deleted = purge_old_records(days)
    return {"deleted": deleted}

Output on success:

INFO  | Task completed | task_name=cleanup status=success duration_ms=142.57 result={'deleted': 83}

Output on failure:

ERROR | Task failed | task_name=cleanup status=failure duration_ms=23.10 error="ConnectionRefusedError..."

Invokes a Python function registered via @task_fn.on_complete. The callback runs synchronously inside the Celery worker after the task finishes. Two callback sources are checked (both are called if present):

  1. The decorator-registered on_complete function
  2. A callable passed at submit time via _on_complete
@task(name="generate_report", notify=["callback"])
def generate_report(report_type: str, date_range: dict) -> dict:
    return {"file": f"/reports/{report_type}.pdf"}

@generate_report.on_complete
def on_report_done(status: str, result: dict, meta: dict):
    if status == "success":
        # Chain: send the report via email
        send_email.submit(
            to=meta.get("requester_email", "admin@example.com"),
            subject=f"Report ready: {result['file']}",
            body="Your report has been generated.",
        )

Publishes a JSON message to a Redis Pub/Sub channel named task_notifications. A listener inside the FastAPI process subscribes and broadcasts through the WebSocketManager, bridging the Celery worker process and the FastAPI application that owns the live WebSocket connections.

@task(name="long_import", notify=["websocket"])
def long_import(file_path: str) -> dict:
    rows = process_csv(file_path)
    return {"imported": rows}

The published message includes routing fields from meta:

{
  "type": "task_completed",
  "task_name": "long_import",
  "status": "success",
  "result": {"imported": 1000},
  "duration_ms": 5432.10,
  "org_id": "org_abc",
  "user_id": "usr_123"
}

Pass org_id and user_id in _meta at submit time to enable targeted delivery to specific WebSocket connections.

Sends an HTTP request (POST or PUT) to an external URL with the task result as a JSON payload. Configuration is provided via the WebhookConfig dataclass.

from singularity.tasks.notifications.webhook_channel import WebhookConfig

slack_config = WebhookConfig(
    url="https://hooks.slack.com/services/T00/B00/xxxxx",
    headers={"Content-Type": "application/json"},
)

@task(
    name="process_order",
    notify=["log", "webhook"],
    notify_config={"webhook": slack_config},
    max_retries=5,
    retry_delay=120,
)
def process_order(order_id: str, items: list) -> dict:
    total = sum(i["price"] for i in items)
    return {"order_id": order_id, "total": total}

WebhookConfig fields:

FieldTypeDefaultDescription
urlstrrequiredThe webhook endpoint URL
headersdict[str, str]{}HTTP headers (Authorization, API keys, etc.)
methodstr"POST"HTTP method -- "POST" or "PUT"
timeoutint10Request timeout in seconds

You can also pass a simple webhook_url in _meta at submit time as a lightweight alternative to WebhookConfig. The channel falls back to this if no notify_config is set.


Submitting Tasks from Services

The TaskRunner is available in every service via acquire.tasks. Call .submit() with the task name and keyword arguments.

from services.__base.service import Service

class ReportsService(Service):
    def __init__(self, acquire):
        super().__init__(acquire)
        self.tasks = acquire.tasks

    async def post(self, report_type: str):
        task_ref = self.tasks.submit(
            "generate_report",
            report_type=report_type,
            date_range={"start": "2024-01-01", "end": "2024-12-31"},
            _meta={"requester_email": "user@example.com"},
        )
        return {"task_id": task_ref.id, "status": "queued"}

submit() Parameters

ParameterTypeDescription
task_namestrThe unique task identifier (must match @task(name=...))
*argspositionalForwarded to the task function
**kwargskeywordForwarded to the task function
_metadictContextual metadata passed to notification channels (not forwarded to the task)
_countdownintDelay in seconds before the task starts executing
_on_completecallableAd-hoc callback invoked on completion (in addition to any @task.on_complete)

submit() validates that all required parameters are satisfied before sending the task to the broker. If a required parameter is missing, it raises a TypeError with a clear description of the expected signature.


Completion Callbacks and Chaining

The callback system enables task chaining without coupling tasks to each other. Register a callback with the @task_fn.on_complete decorator:

@task(name="generate_invoice", notify=["callback", "log"])
def generate_invoice(order_id: str) -> dict:
    pdf_path = create_invoice_pdf(order_id)
    return {"invoice_path": pdf_path, "order_id": order_id}

@generate_invoice.on_complete
def after_invoice(status: str, result: dict, meta: dict):
    """Chain: email the invoice, then notify via Slack."""
    if status == "success":
        # Submit another task -- creates a chain
        send_email.submit(
            to=meta.get("customer_email"),
            subject=f"Invoice for order {result['order_id']}",
            body=f"Your invoice is ready: {result['invoice_path']}",
        )

The callback receives three arguments:

ArgumentTypeDescription
statusstr"success" or "failure"
resultdictThe task's return value (on success) or None (on failure)
metadictThe _meta dict from submit(), plus internal keys like _on_complete_fn

Custom Notification Channels

Extend the notification system by implementing the NotificationChannel abstract base class and registering it with the TaskRunner.

from singularity.tasks.notifications import NotificationChannel

class PagerDutyChannel(NotificationChannel):
    """Alert PagerDuty on task failure."""

    def send(
        self,
        task_name: str,
        status: str,
        result=None,
        error=None,
        duration_ms: float = 0,
        meta=None,
        task_config=None,
    ) -> None:
        if status != "failure":
            return
        # Send PagerDuty alert
        requests.post(
            "https://events.pagerduty.com/v2/enqueue",
            json={
                "routing_key": task_config.get("routing_key"),
                "event_action": "trigger",
                "payload": {
                    "summary": f"Task {task_name} failed: {error}",
                    "severity": "critical",
                    "source": "singularity-tasks",
                },
            },
        )

Register it during application startup:

from singularity.tasks.runner import TaskRunner

runner = TaskRunner()
runner.register_notification_channel("pagerduty", PagerDutyChannel())

Then use it in any task:

@task(
    name="critical_sync",
    notify=["log", "pagerduty"],
    notify_config={"pagerduty": {"routing_key": "R0123456789"}},
)
def critical_sync(source: str) -> dict:
    ...

Retry and Error Handling

Tasks automatically retry on failure up to max_retries times, with retry_delay seconds between each attempt. The Celery worker handles retry scheduling.

@task(
    name="call_external_api",
    max_retries=5,
    retry_delay=120,    # 2 minutes between retries
    time_limit=300,     # kill after 5 minutes
    notify=["log", "webhook"],
    notify_config={"webhook": alert_config},
)
def call_external_api(endpoint: str, payload: dict) -> dict:
    response = requests.post(endpoint, json=payload, timeout=30)
    response.raise_for_status()
    return response.json()

The time_limit is a hard kill. If your task is interrupted by a time limit, the worker terminates the process and the task status becomes failure. Notifications fire in the finally block, so even timed-out tasks dispatch their configured channels.


Worker Management

  1. Start a worker with the default thread pool:

    tasks worker
  2. Increase concurrency for CPU-bound workloads:

    tasks worker --concurrency 8
  3. Switch pool type for I/O-bound tasks:

    tasks worker --pool gevent
  4. Enable the Beat scheduler for periodic tasks:

    tasks worker --beat
  5. Use autoscaling to dynamically adjust the worker count:

    tasks worker --autoscale 10,2

Worker CLI Options

OptionTypeDefaultDescription
--concurrencyint2Number of concurrent worker processes/threads
--loglevelchoiceinfoLogging level: debug, info, warning, error, critical
--queuesstrceleryComma-separated list of queues to consume from
--hostnamestrautoCustom hostname for the worker
--max-tasks-per-childint1000Max tasks before a child process is replaced
--prefetch-multiplierint1Number of messages to prefetch per worker
--poolchoicethreadsPool implementation: prefork, eventlet, gevent, threads, solo
--autoscalestr--Autoscaling range as max,min
--beatflag--Enable the Celery Beat scheduler alongside the worker

Task CLI Commands

The tasks CLI provides commands for listing, inspecting, creating, running, and checking the status of background tasks.

# List all discovered tasks
tasks list

# Detailed info about a specific task (shows parameters, types, defaults)
tasks info send_email

# Create a new task with boilerplate
singularity generate task send_notification -d "Send push notifications to users"

# Trigger a task manually with keyword arguments
tasks run send_email -k '{"to": "user@example.com", "subject": "Hello", "body": "World"}'

# Trigger with a countdown delay (starts in 60 seconds)
tasks run process_order -k '{"order_id": "ord_123", "items": []}' -c 60

# Check the result of a submitted task
tasks result abc12345-task-id

Complete Example

Putting it all together -- a task with webhook notifications, a completion callback, and service-layer submission:

# tasks/executable/process_payment.py

from singularity.tasks import task
from singularity.tasks.notifications.webhook_channel import WebhookConfig

slack_webhook = WebhookConfig(
    url="https://hooks.slack.com/services/T00/B00/xxxxx",
    headers={"Content-Type": "application/json"},
)

@task(
    name="process_payment",
    description="Charge a customer and update their subscription",
    notify=["log", "callback", "webhook"],
    notify_config={"webhook": slack_webhook},
    max_retries=3,
    retry_delay=60,
    time_limit=120,
)
def process_payment(user_id: str, amount: float, currency: str = "usd") -> dict:
    """Charge the customer via Stripe and activate their subscription."""
    charge = stripe.Charge.create(
        amount=int(amount * 100),
        currency=currency,
        customer=user_id,
    )
    return {"charge_id": charge.id, "amount": amount, "user_id": user_id}


@process_payment.on_complete
def on_payment_done(status: str, result: dict, meta: dict):
    if status == "success":
        # Chain: send a receipt email
        from tasks.executable.send_email import send_email
        send_email.submit(
            to=meta.get("customer_email"),
            subject="Payment received",
            body=f"We charged ${result['amount']} to your card.",
        )
# services/v1/billing/service.py

from services.__base.service import Service

class BillingService(Service):
    def __init__(self, acquire):
        super().__init__(acquire)
        self.tasks = acquire.tasks

    async def post(self, user_id: str, amount: float):
        ref = self.tasks.submit(
            "process_payment",
            user_id=user_id,
            amount=amount,
            _meta={
                "customer_email": "customer@example.com",
                "org_id": "org_abc",
            },
        )
        return {"task_id": ref.id, "status": "processing"}