Singularity
Guides

Inter-Service RPC

Understand Singularity's RPC system for local in-process calls and remote cross-deployment communication, including blacklist enforcement, Redis discovery, and the Microservice descriptor pattern.

Singularity provides a built-in RPC (Remote Procedure Call) system that lets services call each other's methods without tight coupling. Calls within the same process are dispatched directly in-memory. Calls to services running on separate deployments are routed transparently over HTTP, discovered via Redis.

The caller always identifies itself by name. The target service can define a blacklist to deny specific callers. From the caller's perspective, the syntax is identical whether the target is local or remote.

Core Concepts

RPCRegistry

Singleton that stores all local and remote service entries. Generates the RPC spec and handles Redis publish/discover.

RPCProxy

Returned by rpc("caller"). Attribute access resolves to a ServiceProxy for the named target service.

ServiceProxy

Dispatches method calls locally (in-process) or remotely (HTTP) with blacklist enforcement.

Microservice

Descriptor for declaring dependencies on services in entirely separate codebases, with per-repo URL and API key.

Making a Local RPC Call

To call another service's method, import rpc and use the chainable proxy syntax. The first argument is the caller's identity (used for blacklist checks). The next attribute is the target service name. The final attribute is the method name.

from singularity.rpc import rpc

class OrdersService:
  def __init__(self, acquire):
    self.acquire = acquire

  async def post(self, data: dict):
    """Create an order and charge the user via RPC."""
    payment = await rpc("orders").payments.post_charge(
      user_id=data["user_id"],
      amount=data["total"],
    )
    return {"order": "confirmed", "payment": payment}

Under the hood, this is what happens:

  1. rpc("orders") creates an RPCProxy bound to the caller identity "orders".
  2. .payments triggers __getattr__, which creates a ServiceProxy for the target "payments".
  3. .post_charge triggers __getattr__ on the ServiceProxy, which looks up "payments" in the registry.
  4. The registry finds a LocalServiceEntry (in-process). The blacklist is checked -- "orders" is not blocked.
  5. The hook-wrapped post_charge endpoint is called directly in memory.
  6. The result is returned to the caller.

Local RPC calls have no network overhead — they are plain Python calls dispatched through the proxy layer, costing one registry lookup and one blacklist check.

Local Calls Run the Full Hook Chain

A local RPC call resolves to the same wrapped callable the HTTP route uses. That means the target's hooks, cache, exception filters, and service_events all run, exactly as they would for an inbound request.

This keeps behavior stable when a service later moves to its own deployment: the call goes from in-process to HTTP, and the semantics do not change.

A consequence worth knowing: if the target declares service_events, an internal RPC post emits created just like an external one. Use a private helper method instead when you need an internal path that skips the chain.

Exposing a Service for RPC

Set rpc_exposed = True on the service class. The Manager collects these during discovery and registers them in the RPCRegistry.

Only endpoint methods are callable over RPC. That means the core verbs (get, post, put, delete, patch) and methods declared in http_exposed. An arbitrary helper method on the class is not reachable — calling it raises RPCMethodNotFound. RPC rides on the same surface your HTTP routes do; it does not expose the whole class.

class PaymentsService:
  """Payment processing service."""

  rpc_exposed = True
  blacklist = ["public", "analytics"]

  # Declare every non-core method you want reachable, over HTTP or RPC.
  http_exposed = ["post=charge", "post=refund"]

  def __init__(self, acquire):
    self.acquire = acquire

  async def post_charge(self, user_id: str, amount: float):
    """Charge a user's payment method."""
    return {"charged": amount, "user": user_id, "transaction_id": "txn_abc123"}

  async def post_refund(self, payment_id: str):
    """Refund a payment."""
    return {"refunded": True, "payment": payment_id}

  async def get(self):
    """List recent payments — also exposed as GET /api/v1/payments."""
    return {"payments": []}

  def _reconcile(self):
    """Internal helper — not an endpoint, so not reachable over RPC."""

All three endpoint methods (post_charge, post_refund, get) become callable via RPC by any service not in the blacklist:

await rpc("orders").payments.post_charge(user_id="u1", amount=9.99)

To make a method reachable over RPC but not over HTTP, declare it in http_exposed and mark it protected:

class UsersService:
  rpc_exposed = True
  http_exposed = ["get=by_email"]
  visibility = {"get_by_email": "protected"}  # RPC only — no HTTP route

  async def get_by_email(self, email: str):
    ...

The Blacklist

The blacklist attribute is a list of caller names that are denied access to this service via RPC. When a blacklisted caller attempts a call, RPCAccessDenied is raised immediately, before the method is invoked.

class PaymentsService:
  rpc_exposed = True
  blacklist = ["public", "analytics"]
  http_exposed = ["post=charge"]

  async def post_charge(self, user_id: str, amount: float):
    return {"charged": amount}

With this configuration:

  • rpc("orders").payments.post_charge(...) succeeds (orders is not blacklisted).
  • rpc("analytics").payments.post_charge(...) raises RPCAccessDenied.
  • rpc("public").payments.post_charge(...) raises RPCAccessDenied.

Blacklist Enforcement Flowchart

Every RPC call passes through this decision tree:

If a service is not found in the local registry or the cached remote registry, the proxy performs an on-demand Redis lookup (discover_single_service) before raising RPCServiceNotFound. This handles the case where a remote service registered after the initial discovery.

Remote RPC (Cross-Deployment)

When multiple Singularity instances run on different hosts, they discover each other through Redis. Each instance publishes its RPC spec to Redis on startup and refreshes it periodically via the heartbeat.

How Remote Calls Work

When the ServiceProxy finds the target in _remote (not _local), it constructs an HTTP request using httpx:

  1. The endpoint URL comes from the remote service's spec (stored in Redis).
  2. The HTTP method matches what the remote service declared (GET, POST, etc.).
  3. An X-RPC-Caller header identifies the calling service, plus X-API-Key when rpc_api_key is set.
  4. Arguments are split between the query string and the JSON body (see below).
  5. The response is parsed as JSON and returned to the caller.

Connections are pooled per timeout, so repeated calls reuse a keep-alive connection instead of re-handshaking.

From the caller's perspective, the code is identical to a local call:

# This works whether payments is local or remote
result = await rpc("orders").payments.post_charge(user_id="abc", amount=9.99)

How Arguments Are Encoded

Arguments are encoded to match how FastAPI reads its own parameters, so a call that works locally lands identically on a remote service.

You callSent as
post_charge(user_id="u1", amount=9.99)Query string ?user_id=u1&amount=9.99
post(data={"title": "x"})JSON body {"title": "x"}
post(data={...}, meta={...})JSON body {"data": {...}, "meta": {...}}
post(user_id="u1", data={...})Query ?user_id=u1 plus JSON body {...}
post(cursor=None)Omitted, so the remote default applies

Scalars (str, int, float, bool) become query parameters; everything else becomes the body. A lone body parameter is sent bare rather than wrapped, because that is how FastAPI reads async def post(self, data: dict). Pydantic models are serialized with model_dump(mode="json").

Both deployments must agree on the API prefix. Endpoint URLs are built from the app's prefix and rpc_base_url. If a caller reaches a deployment mounted on a different prefix, the call 404s — RPCRemoteError says as much in its hint.

Redis Key Structure

Each service is published under a per-service key and a combined spec key:

KeyContentTTL
singularity:rpc:services:{name}Single service spec with base_url, blacklist, methodsrpc_spec_ttl (default 60s)
singularity:rpc:specFull spec for all services on this instancerpc_spec_ttl (default 60s)

All keys for an instance are written in a single pipelined round trip.

The Heartbeat

The Manager starts an asyncio background task that re-publishes the RPC spec to Redis every 30 seconds with a 60-second TTL. This ensures that:

  • If an instance crashes, its keys expire from Redis within 60 seconds.
  • Healthy instances always have fresh entries.
  • New instances can discover existing ones immediately on startup.

Each beat also re-runs discover_remote(), so peers that started since the last beat are picked up and peers whose keys expired are dropped. Without that, a dead deployment would stay in the cache until restart.

Both intervals are configurable:

RPC_HEARTBEAT_INTERVAL=30   # seconds between refreshes
RPC_SPEC_TTL=60             # TTL on published keys — keep it above the interval

To disable the heartbeat (useful during development or testing):

uv run python app.py --dev --no-heartbeat

This sets the SINGULARITY_NO_HEARTBEAT=1 environment variable, which the Manager checks at startup.

Disabling the heartbeat does not disable RPC itself. Local calls still work normally. It only stops the periodic Redis refresh, which means remote services may lose visibility of this instance after the TTL expires.

Securing Inbound RPC

The blacklist check in the sections above runs in the caller's process. That makes it a coupling guard — it stops a service from depending on something it should not — but on its own it is not access control: a remote caller could omit the X-RPC-Caller header, or bypass RPC entirely and call the HTTP route directly.

Singularity therefore re-applies the rules at the receiving end with RPCGuardMiddleware, installed automatically whenever a service is rpc_exposed.

ControlDefaultWhat it does
BlacklistAlways onRejects a request whose X-RPC-Caller is blacklisted with 403
API keyOffRequires a valid X-API-Key on every RPC route, else 401

To lock a mesh down, give every deployment a key and require one:

# On the calling deployment
RPC_API_KEY=shared-secret

# On the receiving deployment
RPC_REQUIRE_API_KEY=true
RPC_API_KEYS=shared-secret,rotating-secret   # comma-separated; defaults to RPC_API_KEY

Outbound calls attach RPC_API_KEY automatically, so once both sides are configured no call sites change.

Because a caller can omit X-RPC-Caller, treat the blacklist as protection against misconfiguration rather than against a hostile client. rpc_require_api_key is the control that actually keeps untrusted clients out. Requests to paths that are not registered RPC services pass through untouched.

The RPC Spec Endpoint

Every Singularity instance exposes its full RPC spec at GET /_rpc/spec. This endpoint is excluded from the OpenAPI schema (include_in_schema=False) and is used for debugging and tooling.

curl http://localhost:8000/_rpc/spec | python -m json.tool

Example response:

{
  "rpc_version": "1.0",
  "base_url": "http://localhost:8000",
  "services": {
    "payments": {
      "blacklist": ["public", "analytics"],
      "methods": {
        "post_charge": {
          "endpoint": "http://localhost:8000/api/v1/payments/charge",
          "http_method": "POST",
          "description": "Charge a user's payment method.",
          "parameters": {
            "user_id": { "type": "str", "required": true },
            "amount": { "type": "float", "required": true }
          },
          "returns": "Any"
        },
        "get": {
          "endpoint": "http://localhost:8000/api/v1/payments",
          "http_method": "GET",
          "description": "List recent payments.",
          "parameters": {},
          "returns": "Any"
        }
      }
    }
  }
}

The spec is generated by RPCRegistry.generate_spec(), which introspects each registered service's methods, extracts their signatures, and produces an OpenAPI-like document.

The Microservice Descriptor

For calling services that live in entirely separate codebases (not just separate instances of the same codebase), use the Microservice descriptor. This provides a clean, class-based interface with per-repo configuration.

from singularity import Microservice
from settings import Settings

settings = Settings()

class BillingMS(Microservice):
  host_url = settings.billing_host_url  # e.g., "http://billing-service:8001"
  api_key = settings.billing_api_key    # sent as X-API-Key header
  timeout = 10                          # request timeout in seconds
  prefix = "/api"                       # API prefix on the remote repo

Use it as a class attribute on your service:

class InvoiceService:
  billing = BillingMS()

  def __init__(self, acquire):
    self.acquire = acquire

  async def post(self, data: dict):
    """Generate an invoice via the billing microservice."""
    # Calls POST http://billing-service:8001/api/invoice
    result = await self.billing.invoice.post(order_id=data["order_id"])
    return {"invoice": result}

How the Descriptor Works

Microservice is a Python descriptor. When accessed on an instance (e.g., self.billing), its __get__ method returns a MicroserviceProxy bound to the caller's identity:

class Microservice:
  host_url: str = ""
  api_key: str = ""
  timeout: float = 30.0
  prefix: str = "/api"
  service_name: str = ""

  def __get__(self, instance, owner):
    if instance is None:
      return self
    caller = getattr(instance, "_service_name", "unknown")
    if self.service_name:
      # Single-service shortcut: skip the repo proxy
      return MicroserviceServiceProxy(
        service_name=self.service_name, host_url=self.host_url, api_key=self.api_key,
        timeout=self.timeout, caller=caller, prefix=self.prefix,
      )
    return MicroserviceProxy(
      host_url=self.host_url, api_key=self.api_key,
      timeout=self.timeout, caller=caller, prefix=self.prefix,
    )

Calls go through the same transport as remote RPC, so argument encoding, connection pooling, and error mapping all behave identically.

Multi-Service vs Single-Service Mode

When service_name is not set, accessing the descriptor returns a MicroserviceProxy. You then specify the target service as the next attribute:

class BillingMS(Microservice):
  host_url = "http://billing:8001"

class OrdersService:
  billing = BillingMS()

  async def checkout(self):
    # billing.invoice -> MicroserviceServiceProxy for "invoice"
    # .post(...) -> POST http://billing:8001/api/invoice
    result = await self.billing.invoice.post(order_id="123")

    # billing.subscription -> MicroserviceServiceProxy for "subscription"
    # .get() -> GET http://billing:8001/api/subscription
    status = await self.billing.subscription.get()

When service_name is set, the descriptor returns a MicroserviceServiceProxy directly, skipping the intermediate repo proxy:

class TestMS(Microservice):
  host_url = "http://test-service:8002"
  service_name = "test"

class ExampleService:
  test = TestMS()

  async def get(self):
    # self.test.get() -> GET http://test-service:8002/api/test
    result = await self.test.get()
    return result

URL Construction Convention

Method names are mapped to HTTP verbs and endpoints using the same convention as the Manager's route registration:

Method callHTTP MethodEndpoint
.get()GET{host_url}{prefix}/{service}
.post(...)POST{host_url}{prefix}/{service}
.get_status(...)GET{host_url}{prefix}/{service}/status
.post_order_payment(...)POST{host_url}{prefix}/{service}/order_payment

Core HTTP verbs (get, post, put, delete, patch) map to the base service endpoint. Prefixed methods (get_xxx, post_xxx) map to sub-paths.

Declarative RPC Dependencies with rpc.remote()

For services that always call the same target, you can use the rpc.remote() descriptor instead of calling rpc() at runtime:

from singularity.rpc import rpc

class OrdersService:
  # Declare dependency at class level
  payments = rpc.remote("payments")

  async def post(self, data: dict):
    # Uses the class attribute — caller identity is auto-injected
    result = await self.payments.post_charge(
      user_id=data["user_id"],
      amount=data["total"],
    )
    return {"order": "confirmed", "payment": result}

The RemoteServiceDescriptor is a Python descriptor that, when accessed on an instance, reads the _service_name attribute (injected by the Manager) and creates a bound ServiceProxy. This is equivalent to calling rpc("orders")["payments"] but declared once at the class level.

Error Handling

The RPC module defines a hierarchy of exceptions that cover every failure mode:

RPCErrorBase class for all RPC errors.
  • RPCServiceNotFoundTarget service is not in the local registry, not in the remote cache, and not discoverable from Redis.
  • RPCMethodNotFoundTarget service exists but does not expose the requested method.
  • RPCAccessDeniedThe caller is in the target service's blacklist.
  • RPCServiceUnavailableThe request never produced a response — host unreachable, connection refused, or timeout.
  • RPCRemoteErrorThe remote service answered with a non-2xx status. Carries status_code and the decoded payload.
from singularity.rpc import (
    rpc,
    RPCServiceNotFound,
    RPCAccessDenied,
    RPCMethodNotFound,
    RPCServiceUnavailable,
    RPCRemoteError,
)

class OrdersService:
  async def post(self, data: dict):
    try:
      result = await rpc("orders").payments.post_charge(
        user_id=data["user_id"],
        amount=data["total"],
      )
      return {"order": "confirmed", "payment": result}

    except RPCAccessDenied:
      return {"error": "Orders service is not allowed to access payments"}

    except RPCServiceNotFound:
      return {"error": "Payments service is not registered or unreachable"}

    except RPCMethodNotFound:
      return {"error": "post_charge is not exposed on payments"}

    except RPCServiceUnavailable as e:
      return {"error": f"Payments service is down: {e}"}

    except RPCRemoteError as e:
      # The call reached payments, which rejected it.
      return {"error": f"Payments returned {e.status_code}", "detail": e.payload}

Every failure mode surfaces as an RPCError subclass — httpx exceptions never escape the transport layer, so except RPCError is sufficient to catch anything the call can raise. RPCRemoteError exposes status_code and the decoded payload, and its .hint explains the common causes of 401, 403, 404, and 422.

Startup Sequence

Understanding when RPC initialization happens helps with debugging:

  1. Module load: Manager.register_services() scans for services. For each one it wraps every non-private endpoint with the hook chain, mounts the public ones as HTTP routes, and collects rpc_exposed services into _rpc_pending along with their wrapped callables.
  2. RPC init: _init_rpc() calls registry.register(...) for each, passing the app's prefix so published endpoint URLs match the mounted routes. RPCGuardMiddleware is installed.
  3. Lifespan startup: manager.startup() is called by FastAPI's lifespan:
    • registry.publish_to_redis(ttl=rpc_spec_ttl) publishes each service's spec as a Redis key.
    • registry.discover_remote() scans Redis for keys matching singularity:rpc:services:* and populates _remote.
    • If the heartbeat is enabled and RPC services exist, _heartbeat_loop is started as an asyncio task.
  4. Serving: The app accepts requests. RPC calls work for both local and discovered remote services. Each heartbeat republishes and re-scans.
  5. Lifespan shutdown: manager.shutdown() cancels the heartbeat, calls registry.remove_from_redis(), and closes the pooled Redis and HTTP connections.

Full Working Example

Here is a complete example with two services that communicate via RPC:

# services/v1/payments/service.py
"""Payment processing service."""

from singularity.core.acquire import Acquire


class PaymentsService:
  """Handles charging and refunding payments."""

  rpc_exposed = True
  blacklist = ["public"]
  http_exposed = ["post=charge", "post=refund"]

  def __init__(self, acquire: Acquire):
    self.acquire = acquire
    self.db = acquire.db_session

  async def get(self):
    """List recent payments."""
    return {"payments": []}

  async def post_charge(self, user_id: str, amount: float):
    """Charge a user's payment method."""
    async with self.db() as session:
      # Record the payment
      payment = Payment(user_id=user_id, amount=amount, status="charged")
      session.add(payment)
      await session.commit()
      return {
        "charged": amount,
        "user": user_id,
        "transaction_id": payment.id,
      }

  async def post_refund(self, payment_id: str):
    """Refund a payment."""
    return {"refunded": True, "payment": payment_id}
# services/v1/orders/service.py
"""Order management service."""

from singularity.core.acquire import Acquire
from singularity.rpc import rpc


class OrdersService:
  """Creates orders and coordinates payment via RPC."""

  rpc_exposed = True

  def __init__(self, acquire: Acquire):
    self.acquire = acquire

  async def get(self):
    """List orders."""
    return {"orders": []}

  async def post(self, data: dict):
    """Create an order and charge the user."""
    payment = await rpc("orders").payments.post_charge(
      user_id=data["user_id"],
      amount=data["total"],
    )
    return {
      "order_id": "ord_new",
      "status": "confirmed",
      "payment": payment,
    }

Scaffold both with:

singularity generate service v1/payments -d "Payment processing" --template crud --rpc --db
singularity generate service v1/orders -d "Order management" --template crud --rpc