Inter-Service RPC
Understand Singularity's RPC system for local in-process calls and remote cross-deployment communication, including blacklist enforcement, Redis discovery, and the Microservice descriptor pattern.
Singularity provides a built-in RPC (Remote Procedure Call) system that lets services call each other's methods without tight coupling. Calls within the same process are dispatched directly in-memory. Calls to services running on separate deployments are routed transparently over HTTP, discovered via Redis.
The caller always identifies itself by name. The target service can define a blacklist to deny specific callers. From the caller's perspective, the syntax is identical whether the target is local or remote.
Core Concepts
RPCRegistry
Singleton that stores all local and remote service entries. Generates the RPC spec and handles Redis publish/discover.
RPCProxy
Returned by rpc("caller"). Attribute access resolves to a ServiceProxy for the named target service.
ServiceProxy
Dispatches method calls locally (in-process) or remotely (HTTP) with blacklist enforcement.
Microservice
Descriptor for declaring dependencies on services in entirely separate codebases, with per-repo URL and API key.
Making a Local RPC Call
To call another service's method, import rpc and use the chainable proxy syntax. The first argument is the caller's identity (used for blacklist checks). The next attribute is the target service name. The final attribute is the method name.
from singularity.rpc import rpc
class OrdersService:
def __init__(self, acquire):
self.acquire = acquire
async def post(self, data: dict):
"""Create an order and charge the user via RPC."""
payment = await rpc("orders").payments.post_charge(
user_id=data["user_id"],
amount=data["total"],
)
return {"order": "confirmed", "payment": payment}Under the hood, this is what happens:
rpc("orders")creates anRPCProxybound to the caller identity"orders"..paymentstriggers__getattr__, which creates aServiceProxyfor the target"payments"..post_chargetriggers__getattr__on theServiceProxy, which looks up"payments"in the registry.- The registry finds a
LocalServiceEntry(in-process). The blacklist is checked --"orders"is not blocked. - The hook-wrapped
post_chargeendpoint is called directly in memory. - The result is returned to the caller.
Local RPC calls have no network overhead — they are plain Python calls dispatched through the proxy layer, costing one registry lookup and one blacklist check.
Local Calls Run the Full Hook Chain
A local RPC call resolves to the same wrapped callable the HTTP route uses. That means the target's hooks, cache, exception filters, and service_events all run, exactly as they would for an inbound request.
This keeps behavior stable when a service later moves to its own deployment: the call goes from in-process to HTTP, and the semantics do not change.
A consequence worth knowing: if the target declares service_events, an internal RPC post emits created just like an external one. Use a private helper method instead when you need an internal path that skips the chain.
Exposing a Service for RPC
Set rpc_exposed = True on the service class. The Manager collects these during discovery and registers them in the RPCRegistry.
Only endpoint methods are callable over RPC. That means the core verbs (get, post, put, delete, patch) and methods declared in http_exposed. An arbitrary helper method on the class is not reachable — calling it raises RPCMethodNotFound. RPC rides on the same surface your HTTP routes do; it does not expose the whole class.
class PaymentsService:
"""Payment processing service."""
rpc_exposed = True
blacklist = ["public", "analytics"]
# Declare every non-core method you want reachable, over HTTP or RPC.
http_exposed = ["post=charge", "post=refund"]
def __init__(self, acquire):
self.acquire = acquire
async def post_charge(self, user_id: str, amount: float):
"""Charge a user's payment method."""
return {"charged": amount, "user": user_id, "transaction_id": "txn_abc123"}
async def post_refund(self, payment_id: str):
"""Refund a payment."""
return {"refunded": True, "payment": payment_id}
async def get(self):
"""List recent payments — also exposed as GET /api/v1/payments."""
return {"payments": []}
def _reconcile(self):
"""Internal helper — not an endpoint, so not reachable over RPC."""All three endpoint methods (post_charge, post_refund, get) become callable via RPC by any service not in the blacklist:
await rpc("orders").payments.post_charge(user_id="u1", amount=9.99)To make a method reachable over RPC but not over HTTP, declare it in http_exposed and mark it protected:
class UsersService:
rpc_exposed = True
http_exposed = ["get=by_email"]
visibility = {"get_by_email": "protected"} # RPC only — no HTTP route
async def get_by_email(self, email: str):
...The Blacklist
The blacklist attribute is a list of caller names that are denied access to this service via RPC. When a blacklisted caller attempts a call, RPCAccessDenied is raised immediately, before the method is invoked.
class PaymentsService:
rpc_exposed = True
blacklist = ["public", "analytics"]
http_exposed = ["post=charge"]
async def post_charge(self, user_id: str, amount: float):
return {"charged": amount}With this configuration:
rpc("orders").payments.post_charge(...)succeeds (orders is not blacklisted).rpc("analytics").payments.post_charge(...)raisesRPCAccessDenied.rpc("public").payments.post_charge(...)raisesRPCAccessDenied.
Blacklist Enforcement Flowchart
Every RPC call passes through this decision tree:
If a service is not found in the local registry or the cached remote registry, the proxy performs an on-demand Redis lookup (discover_single_service) before raising RPCServiceNotFound. This handles the case where a remote service registered after the initial discovery.
Remote RPC (Cross-Deployment)
When multiple Singularity instances run on different hosts, they discover each other through Redis. Each instance publishes its RPC spec to Redis on startup and refreshes it periodically via the heartbeat.
How Remote Calls Work
When the ServiceProxy finds the target in _remote (not _local), it constructs an HTTP request using httpx:
- The endpoint URL comes from the remote service's spec (stored in Redis).
- The HTTP method matches what the remote service declared (GET, POST, etc.).
- An
X-RPC-Callerheader identifies the calling service, plusX-API-Keywhenrpc_api_keyis set. - Arguments are split between the query string and the JSON body (see below).
- The response is parsed as JSON and returned to the caller.
Connections are pooled per timeout, so repeated calls reuse a keep-alive connection instead of re-handshaking.
From the caller's perspective, the code is identical to a local call:
# This works whether payments is local or remote
result = await rpc("orders").payments.post_charge(user_id="abc", amount=9.99)How Arguments Are Encoded
Arguments are encoded to match how FastAPI reads its own parameters, so a call that works locally lands identically on a remote service.
| You call | Sent as |
|---|---|
post_charge(user_id="u1", amount=9.99) | Query string ?user_id=u1&amount=9.99 |
post(data={"title": "x"}) | JSON body {"title": "x"} |
post(data={...}, meta={...}) | JSON body {"data": {...}, "meta": {...}} |
post(user_id="u1", data={...}) | Query ?user_id=u1 plus JSON body {...} |
post(cursor=None) | Omitted, so the remote default applies |
Scalars (str, int, float, bool) become query parameters; everything else becomes the body. A lone body parameter is sent bare rather than wrapped, because that is how FastAPI reads async def post(self, data: dict). Pydantic models are serialized with model_dump(mode="json").
Both deployments must agree on the API prefix. Endpoint URLs are built from the app's prefix and rpc_base_url. If a caller reaches a deployment mounted on a different prefix, the call 404s — RPCRemoteError says as much in its hint.
Redis Key Structure
Each service is published under a per-service key and a combined spec key:
| Key | Content | TTL |
|---|---|---|
singularity:rpc:services:{name} | Single service spec with base_url, blacklist, methods | rpc_spec_ttl (default 60s) |
singularity:rpc:spec | Full spec for all services on this instance | rpc_spec_ttl (default 60s) |
All keys for an instance are written in a single pipelined round trip.
The Heartbeat
The Manager starts an asyncio background task that re-publishes the RPC spec to Redis every 30 seconds with a 60-second TTL. This ensures that:
- If an instance crashes, its keys expire from Redis within 60 seconds.
- Healthy instances always have fresh entries.
- New instances can discover existing ones immediately on startup.
Each beat also re-runs discover_remote(), so peers that started since the last beat are picked up and peers whose keys expired are dropped. Without that, a dead deployment would stay in the cache until restart.
Both intervals are configurable:
RPC_HEARTBEAT_INTERVAL=30 # seconds between refreshes
RPC_SPEC_TTL=60 # TTL on published keys — keep it above the intervalTo disable the heartbeat (useful during development or testing):
uv run python app.py --dev --no-heartbeatThis sets the SINGULARITY_NO_HEARTBEAT=1 environment variable, which the Manager checks at startup.
Disabling the heartbeat does not disable RPC itself. Local calls still work normally. It only stops the periodic Redis refresh, which means remote services may lose visibility of this instance after the TTL expires.
Securing Inbound RPC
The blacklist check in the sections above runs in the caller's process. That makes it a coupling guard — it stops a service from depending on something it should not — but on its own it is not access control: a remote caller could omit the X-RPC-Caller header, or bypass RPC entirely and call the HTTP route directly.
Singularity therefore re-applies the rules at the receiving end with RPCGuardMiddleware, installed automatically whenever a service is rpc_exposed.
| Control | Default | What it does |
|---|---|---|
| Blacklist | Always on | Rejects a request whose X-RPC-Caller is blacklisted with 403 |
| API key | Off | Requires a valid X-API-Key on every RPC route, else 401 |
To lock a mesh down, give every deployment a key and require one:
# On the calling deployment
RPC_API_KEY=shared-secret
# On the receiving deployment
RPC_REQUIRE_API_KEY=true
RPC_API_KEYS=shared-secret,rotating-secret # comma-separated; defaults to RPC_API_KEYOutbound calls attach RPC_API_KEY automatically, so once both sides are configured no call sites change.
Because a caller can omit X-RPC-Caller, treat the blacklist as protection against misconfiguration rather than against a hostile client. rpc_require_api_key is the control that actually keeps untrusted clients out. Requests to paths that are not registered RPC services pass through untouched.
The RPC Spec Endpoint
Every Singularity instance exposes its full RPC spec at GET /_rpc/spec. This endpoint is excluded from the OpenAPI schema (include_in_schema=False) and is used for debugging and tooling.
curl http://localhost:8000/_rpc/spec | python -m json.toolExample response:
{
"rpc_version": "1.0",
"base_url": "http://localhost:8000",
"services": {
"payments": {
"blacklist": ["public", "analytics"],
"methods": {
"post_charge": {
"endpoint": "http://localhost:8000/api/v1/payments/charge",
"http_method": "POST",
"description": "Charge a user's payment method.",
"parameters": {
"user_id": { "type": "str", "required": true },
"amount": { "type": "float", "required": true }
},
"returns": "Any"
},
"get": {
"endpoint": "http://localhost:8000/api/v1/payments",
"http_method": "GET",
"description": "List recent payments.",
"parameters": {},
"returns": "Any"
}
}
}
}
}The spec is generated by RPCRegistry.generate_spec(), which introspects each registered service's methods, extracts their signatures, and produces an OpenAPI-like document.
The Microservice Descriptor
For calling services that live in entirely separate codebases (not just separate instances of the same codebase), use the Microservice descriptor. This provides a clean, class-based interface with per-repo configuration.
from singularity import Microservice
from settings import Settings
settings = Settings()
class BillingMS(Microservice):
host_url = settings.billing_host_url # e.g., "http://billing-service:8001"
api_key = settings.billing_api_key # sent as X-API-Key header
timeout = 10 # request timeout in seconds
prefix = "/api" # API prefix on the remote repoUse it as a class attribute on your service:
class InvoiceService:
billing = BillingMS()
def __init__(self, acquire):
self.acquire = acquire
async def post(self, data: dict):
"""Generate an invoice via the billing microservice."""
# Calls POST http://billing-service:8001/api/invoice
result = await self.billing.invoice.post(order_id=data["order_id"])
return {"invoice": result}How the Descriptor Works
Microservice is a Python descriptor. When accessed on an instance (e.g., self.billing), its __get__ method returns a MicroserviceProxy bound to the caller's identity:
class Microservice:
host_url: str = ""
api_key: str = ""
timeout: float = 30.0
prefix: str = "/api"
service_name: str = ""
def __get__(self, instance, owner):
if instance is None:
return self
caller = getattr(instance, "_service_name", "unknown")
if self.service_name:
# Single-service shortcut: skip the repo proxy
return MicroserviceServiceProxy(
service_name=self.service_name, host_url=self.host_url, api_key=self.api_key,
timeout=self.timeout, caller=caller, prefix=self.prefix,
)
return MicroserviceProxy(
host_url=self.host_url, api_key=self.api_key,
timeout=self.timeout, caller=caller, prefix=self.prefix,
)Calls go through the same transport as remote RPC, so argument encoding, connection pooling, and error mapping all behave identically.
Multi-Service vs Single-Service Mode
When service_name is not set, accessing the descriptor returns a MicroserviceProxy. You then specify the target service as the next attribute:
class BillingMS(Microservice):
host_url = "http://billing:8001"
class OrdersService:
billing = BillingMS()
async def checkout(self):
# billing.invoice -> MicroserviceServiceProxy for "invoice"
# .post(...) -> POST http://billing:8001/api/invoice
result = await self.billing.invoice.post(order_id="123")
# billing.subscription -> MicroserviceServiceProxy for "subscription"
# .get() -> GET http://billing:8001/api/subscription
status = await self.billing.subscription.get()When service_name is set, the descriptor returns a MicroserviceServiceProxy directly, skipping the intermediate repo proxy:
class TestMS(Microservice):
host_url = "http://test-service:8002"
service_name = "test"
class ExampleService:
test = TestMS()
async def get(self):
# self.test.get() -> GET http://test-service:8002/api/test
result = await self.test.get()
return resultURL Construction Convention
Method names are mapped to HTTP verbs and endpoints using the same convention as the Manager's route registration:
| Method call | HTTP Method | Endpoint |
|---|---|---|
.get() | GET | {host_url}{prefix}/{service} |
.post(...) | POST | {host_url}{prefix}/{service} |
.get_status(...) | GET | {host_url}{prefix}/{service}/status |
.post_order_payment(...) | POST | {host_url}{prefix}/{service}/order_payment |
Core HTTP verbs (get, post, put, delete, patch) map to the base service endpoint. Prefixed methods (get_xxx, post_xxx) map to sub-paths.
Declarative RPC Dependencies with rpc.remote()
For services that always call the same target, you can use the rpc.remote() descriptor instead of calling rpc() at runtime:
from singularity.rpc import rpc
class OrdersService:
# Declare dependency at class level
payments = rpc.remote("payments")
async def post(self, data: dict):
# Uses the class attribute — caller identity is auto-injected
result = await self.payments.post_charge(
user_id=data["user_id"],
amount=data["total"],
)
return {"order": "confirmed", "payment": result}The RemoteServiceDescriptor is a Python descriptor that, when accessed on an instance, reads the _service_name attribute (injected by the Manager) and creates a bound ServiceProxy. This is equivalent to calling rpc("orders")["payments"] but declared once at the class level.
Error Handling
The RPC module defines a hierarchy of exceptions that cover every failure mode:
RPCErrorBase class for all RPC errors.RPCServiceNotFoundTarget service is not in the local registry, not in the remote cache, and not discoverable from Redis.RPCMethodNotFoundTarget service exists but does not expose the requested method.RPCAccessDeniedThe caller is in the target service's blacklist.RPCServiceUnavailableThe request never produced a response — host unreachable, connection refused, or timeout.RPCRemoteErrorThe remote service answered with a non-2xx status. Carries status_code and the decoded payload.
from singularity.rpc import (
rpc,
RPCServiceNotFound,
RPCAccessDenied,
RPCMethodNotFound,
RPCServiceUnavailable,
RPCRemoteError,
)
class OrdersService:
async def post(self, data: dict):
try:
result = await rpc("orders").payments.post_charge(
user_id=data["user_id"],
amount=data["total"],
)
return {"order": "confirmed", "payment": result}
except RPCAccessDenied:
return {"error": "Orders service is not allowed to access payments"}
except RPCServiceNotFound:
return {"error": "Payments service is not registered or unreachable"}
except RPCMethodNotFound:
return {"error": "post_charge is not exposed on payments"}
except RPCServiceUnavailable as e:
return {"error": f"Payments service is down: {e}"}
except RPCRemoteError as e:
# The call reached payments, which rejected it.
return {"error": f"Payments returned {e.status_code}", "detail": e.payload}Every failure mode surfaces as an RPCError subclass — httpx exceptions never escape the transport layer, so except RPCError is sufficient to catch anything the call can raise. RPCRemoteError exposes status_code and the decoded payload, and its .hint explains the common causes of 401, 403, 404, and 422.
Startup Sequence
Understanding when RPC initialization happens helps with debugging:
- Module load:
Manager.register_services()scans for services. For each one it wraps every non-private endpoint with the hook chain, mounts the public ones as HTTP routes, and collectsrpc_exposedservices into_rpc_pendingalong with their wrapped callables. - RPC init:
_init_rpc()callsregistry.register(...)for each, passing the app'sprefixso published endpoint URLs match the mounted routes.RPCGuardMiddlewareis installed. - Lifespan startup:
manager.startup()is called by FastAPI's lifespan:registry.publish_to_redis(ttl=rpc_spec_ttl)publishes each service's spec as a Redis key.registry.discover_remote()scans Redis for keys matchingsingularity:rpc:services:*and populates_remote.- If the heartbeat is enabled and RPC services exist,
_heartbeat_loopis started as an asyncio task.
- Serving: The app accepts requests. RPC calls work for both local and discovered remote services. Each heartbeat republishes and re-scans.
- Lifespan shutdown:
manager.shutdown()cancels the heartbeat, callsregistry.remove_from_redis(), and closes the pooled Redis and HTTP connections.
Full Working Example
Here is a complete example with two services that communicate via RPC:
# services/v1/payments/service.py
"""Payment processing service."""
from singularity.core.acquire import Acquire
class PaymentsService:
"""Handles charging and refunding payments."""
rpc_exposed = True
blacklist = ["public"]
http_exposed = ["post=charge", "post=refund"]
def __init__(self, acquire: Acquire):
self.acquire = acquire
self.db = acquire.db_session
async def get(self):
"""List recent payments."""
return {"payments": []}
async def post_charge(self, user_id: str, amount: float):
"""Charge a user's payment method."""
async with self.db() as session:
# Record the payment
payment = Payment(user_id=user_id, amount=amount, status="charged")
session.add(payment)
await session.commit()
return {
"charged": amount,
"user": user_id,
"transaction_id": payment.id,
}
async def post_refund(self, payment_id: str):
"""Refund a payment."""
return {"refunded": True, "payment": payment_id}# services/v1/orders/service.py
"""Order management service."""
from singularity.core.acquire import Acquire
from singularity.rpc import rpc
class OrdersService:
"""Creates orders and coordinates payment via RPC."""
rpc_exposed = True
def __init__(self, acquire: Acquire):
self.acquire = acquire
async def get(self):
"""List orders."""
return {"orders": []}
async def post(self, data: dict):
"""Create an order and charge the user."""
payment = await rpc("orders").payments.post_charge(
user_id=data["user_id"],
amount=data["total"],
)
return {
"order_id": "ord_new",
"status": "confirmed",
"payment": payment,
}Scaffold both with:
singularity generate service v1/payments -d "Payment processing" --template crud --rpc --db
singularity generate service v1/orders -d "Order management" --template crud --rpc