Loomscope

ADR-0001: REST and SSE instead of gRPC for daemon transport

Edit this page

Status: Accepted Date: 2026-01-15

Context

Loomscope splits into a control plane and one or more daemons that scan networks. The daemons live wherever the networks are: a branch office, a DMZ, an air-gapped rack. They need to receive scan jobs, stream observations back, and heartbeat. The control plane needs to push updates to a browser as they arrive.

The obvious candidate for the daemon link is gRPC. It gives bidirectional streaming, a schema-first contract with generated clients on both ends, and efficient binary framing. Scanopy — the system Loomscope learns from — uses REST and SSE, and has been criticised for it.

Three properties constrain the choice:

  1. Daemons sit behind networks we do not control. Corporate proxies, TLS-terminating load balancers and firewalls handle HTTP/1.1 well and HTTP/2 with trailers — which gRPC requires — inconsistently.
  2. The traffic shape is modest. A daemon polls for jobs and posts batched observations. This is not a high-frequency RPC workload; it is a handful of requests per minute per daemon carrying a few hundred kilobytes.
  3. The operator installing this is not a platform team. Loomscope targets organisations that run a Compose file. Every additional protocol is something they have to debug when it breaks.

Decision

The daemon talks to the control plane over HTTP REST under /api/v1/. Browser updates use Server-Sent Events backed by PostgreSQL LISTEN/NOTIFY.

The contract is still schema-first: Zod schemas in packages/contracts are the source of truth, generating an OpenAPI 3.1 document, which generates the daemon's Go client through oapi-codegen. We keep the property that made gRPC attractive — a single definition producing both ends — without adopting its transport.

Job delivery uses HTTP long-polling (daemon_poll), with a server_poll mode where the control plane pushes to the daemon instead, for DMZ deployments where inbound connections to the daemon are the only ones permitted.

Alternatives considered

gRPC with bidirectional streaming. Rejected. It solves a scaling problem Loomscope does not have, and creates deployment problems it would. gRPC through a corporate proxy is a support burden; gRPC-Web adds a translating proxy component; and debugging becomes impossible with curl, which matters more than it sounds when a customer's daemon will not connect and you are on a call with their network team.

WebSockets. Rejected. Bidirectional, which suits job push, but stateful connections complicate horizontal scaling of the control plane and add reconnection logic on both ends. SSE is unidirectional and that is sufficient: the browser only receives.

MQTT or another broker-mediated protocol. Rejected together with the broker itself — see ADR-0002.

Consequences

Accepted costs.

  • Long-polling holds a connection per idle daemon. Daemon traffic runs on a dedicated PostgreSQL pool so a poll storm cannot starve UI queries, but the ceiling on concurrent daemons is lower than a streaming protocol would give.
  • JSON over HTTP is more verbose than protobuf. At Loomscope's message volume this costs bandwidth we can afford.
  • We maintain the OpenAPI generation pipeline ourselves rather than getting it from the RPC framework.

Gains.

  • A daemon can be debugged with curl, and its traffic read in any proxy.
  • No HTTP/2 requirement anywhere in the path.
  • The same REST surface that serves daemons becomes the public API (ADR-0003) at close to zero additional cost.

Boundary. If a deployment ever needs thousands of daemons against one control plane, this decision should be revisited for the job-distribution path specifically — not for the whole API.