Developer guide

For people changing Loomscope's code. CONTRIBUTING.md covers setup and the pull-request rules; this guide covers how the system is put together and how to extend it.

Contents

• Architecture
• Repository layout
• The request path
• Tenant isolation
• The contract pipeline
• Adding a service signature
• Adding a scanner
• Adding a topology view
• Background jobs
• Realtime
• Database conventions
• Testing
• Known rough edges

Architecture

Loomscope is a control plane plus scanners. Its shape follows from three decisions worth reading before changing anything structural:

• ADR-0001 — REST and SSE rather than gRPC, because daemons live behind networks we do not control.
• ADR-0002 — PostgreSQL is the only datastore. No Redis, no broker.
• ADR-0007 — where the UI styling is going, and why it is currently inconsistent.

The daemon never accepts inbound connections in the default mode. It polls for jobs and posts observations, so it works from a branch office behind NAT with no firewall changes.

Repository layout

apps/server            Control plane — Next.js 15 App Router
  app/(app)/           Authenticated UI pages
  app/api/v1/          REST API: daemon endpoints and admin routes
  server/db/           Drizzle schema, migrations, scoped transactions
  server/topology/     The four topology generators, layout, persistence
  server/cloud/        AWS and Kubernetes collectors
  server/cve/          CVE matching
  server/snapshot/     Capture, diff, PDF attestation
  server/ai/           Assistant provider abstraction and tools
  server/auth/         Session, org scope, per-site permissions
apps/docs              Nextra documentation site
services/daemon        Go scanner
  internal/scan/       ICMP, ARP, TCP, UDP, DNS, TLS, SNMP, Docker
  internal/signatures/ Pattern engine, built-in signatures, YAML loader
  internal/netflow/    NetFlow v5/v9, IPFIX, sFlow parsers
services/cve-ingest    Optional standalone OSV/NVD mirror worker
packages/contracts     Zod schemas → OpenAPI → generated Go client
packages/shared        Shared TypeScript types
infra                  Compose files, Dockerfiles, systemd units

The request path

A discovery observation travels like this:

1. The daemon polls /api/v1/daemon/jobs/poll and receives a scan job.
2. It scans, batching results in internal/discovery/batcher.go.
3. It posts to /api/v1/daemon/observations.
4. apps/server/app/api/v1/daemon/observations/route.ts authenticates the daemon, opens an org-scoped transaction, and upserts hosts, ports, services, interfaces, certificates.
5. The upsert emits NOTIFY, which the SSE endpoint relays to open browsers.

That observations route is the hot path and the most consequential file in the control plane. It is also, currently, untested — see Known rough edges.

Tenant isolation

This is the part to understand before writing any query.

Every business table carries organization_id and has row-level security enabled and FORCEd. 0001_init.sql applies this in a loop over every table with that column, so new tables are covered automatically by having the column.

Access goes through a scoped transaction:

await withSessionOrg(async (tx, orgId) => {
  // SET LOCAL app.org_id has already run on this transaction.
  // RLS filters everything, even if this query forgets its WHERE clause.
});

Three wrappers exist:

Two tables reach their tenant through a parent rather than a column — membership_site_scopes and snapshot_entities. Both have an explicit policy that joins to the parent. If you add a child table without organization_id, you must write that policy; the loop will not cover you.

Never bypass this for an admin endpoint. Cross-org access requires an explicit superadmin scope and an audit entry.

The contract pipeline

packages/contracts holds Zod schemas and is the source of truth.

Zod schemas  →  OpenAPI 3.1  →  Go client
   (hand)       make openapi    make oapi-go

Changing a daemon-facing schema means: edit the Zod schema, run make openapi && make oapi-go, and commit the regenerated artefacts. A pre-commit hook does this via lint-staged; if you bypass hooks, do it by hand. A stale openapi.json means the published contract lies about the API.

Adding a service signature

Two paths, producing the same internal type.

Built-in, compiled into the daemon, for common services:

// services/daemon/internal/signatures/core/grafana.go
func (GrafanaSignature) ID() string { return "grafana" }
func (GrafanaSignature) DiscoveryPattern() pattern.Pattern {
    return pattern.Endpoint{
        Port:   pattern.PortHTTP,
        Path:   "/api/health",
        Expect: "grafana",
    }
}
func init() { registry.Register(GrafanaSignature{}) }

Custom, YAML loaded at boot and on SIGHUP, for anything in-house:

version: 1
service:
  id: internal-auth-service
  name: Internal Auth Service
  category: Web
  confidence_default: 85
  pattern:
    all_of:
      - port: { protocol: tcp, number: 8443 }
      - endpoint:
          {
            port: { protocol: tcp, number: 8443 },
            tls: true,
            path: /healthz,
            expect: "auth-service v",
          }

The pattern grammar is identical in both: port, endpoint, all_of, any_of, not, is_gateway, mac_vendor, subnet_type, none. The YAML loader validates strictly — an unknown field or bad enum fails at boot with a structured error rather than being ignored.

Prefer an endpoint match over a bare port. A port number is a guess; a response fragment is evidence, and the confidence score you assign propagates into CVE matching.

Adding a scanner

Scanners live in services/daemon/internal/scan/<name>/. The contract:

• Take ctx context.Context first, and honour cancellation. Scans get cancelled.
• Never assume IPv4. Use netip.Addr throughout and test both families.
• Emit through the batcher rather than posting directly — it handles batching and backpressure.
• Log with zerolog, structured fields, never string interpolation, never a credential.
• Provide a no-op implementation for platforms you do not support, behind a build tag, as arp_other.go does.

Then extend the observation contract in packages/contracts, regenerate, and handle the new fields in the observations route.

Adding a topology view

Generators live in server/topology/ and share a shape:

export async function generateX(
  tx: UiTx,
  siteId: string | null,
): Promise<TopologyPayload>;

Return nodes and edges; persist.ts handles upserting, preserving user-moved positions, and layout. Two constraints that are easy to get wrong:

• groupId is a uuid foreign key to a parent node, not a label. It is cast to ::uuid on insert, so a prefixed string like site:abc throws. Non-structural grouping information belongs in node metadata.
• Layout engines differ: dagre for hierarchical views (L2, workloads), elk for graph-shaped ones (L3, application). elk ignores groupId entirely.

Background jobs

Schedulers start on the first request through the (app) layout, guarded by a singleton. They run in-process:

They use withForcedOrg because they have no session. In-process scheduling means multiple replicas would each run them — a real limitation, and the reason job claiming uses FOR UPDATE SKIP LOCKED.

Realtime

PostgreSQL NOTIFY → an SSE endpoint → the browser. Payloads are capped at 8000 bytes, so notifications carry identifiers and the client re-reads. Do not add websockets (ADR-0001).

Database conventions

• Migrations are SQL, forward-only, numbered, and never edited once shipped.
• SET lock_timeout = '3s' at the top of any migration that runs ALTER.
• Expand-and-contract for renames, type changes and new NOT NULL columns.
• Backfills batch 5000 rows, in their own migration.
• CREATE INDEX CONCURRENTLY is not available — the runner wraps each file in an implicit transaction. If a large live table needs an index, that migration has to be run outside the normal runner, deliberately.
• No raw SQL in business code without going through a scoped transaction.

squawk lints the migrations a change adds. Two rules are disabled for the reasons recorded in squawk.toml.

Testing

make typecheck     # tsc --noEmit across all packages
make test          # vitest + go test -race

What exists: 596 TypeScript tests over parsing, diffing, CVE matching, subnet derivation, permissions, export serialisation, the public REST collections and the SCIM surface; Go tests over the pattern engine, SNMP table parsers, NetFlow parsers and the YAML loader; shell E2E scripts under scripts/ that run against a live stack.

Many of those tests need a database and skip themselves when DATABASE_URL is absent — a suite that reports green having run a third of itself. apps/server/vitest.config.ts loads the repo's .env, so this only bites if that file is missing. If the count is far below 596, that is why.

Where to put a new test:

• Pure logic → a __tests__ directory next to the code.
• Anything touching the database → an integration test using the real PostgreSQL that CI provides. Do not mock Drizzle.
• Go → _test.go beside the file, with -race if goroutines are involved.

On Windows

The stack is developed under WSL2, and three things go wrong there in ways that do not name their cause: WSL inherits the Windows PATH so pnpm resolves to a Windows executable and dies with This: not found; nvm is only sourced by an interactive shell, so wsl bash -c has no node; and the root .env is not read by Next.js, which only looks in apps/server/.

scripts/wsl-run.sh handles all three:

wsl bash scripts/wsl-run.sh pnpm typecheck
wsl bash scripts/wsl-run.sh make test

One further trap it cannot fix: next dev running in WSL against a /mnt/… path gets no inotify events for edits made from Windows, so it never hot-reloads. It will keep serving the previous build while the file on disk says otherwise. Restart it after editing from the Windows side.

Known rough edges

Stated so you are not surprised, and because these are good first contributions.

231 unchecked casts. Raw SQL results are cast with as unknown as RowType, bypassing both Zod and the type checker. A wrong column name fails at runtime, not compile time. The count has grown, not shrunk — every new raw query adds one. apps/server/server/rest/collection.ts shows the shape a fix would take: one place that knows how a row becomes a wire object, rather than a cast per endpoint.

21 of 28 API routes are covered. The eight public collections, the five daemon endpoints, /api/v1/metrics, and the session-authenticated routes — the last by one suite asserting the property that matters uniformly: a caller the route cannot identify is refused, with a response rather than a thrown error. What remains uncovered is third-party adapters (Better-Auth's catch-all, the tRPC handler) and the SCIM surface, which scripts/scim-e2e.sh exercises against a running server instead.

Browser tests do not run here. Playwright is installed and specs exist, but playwright install --with-deps chromium needs root for the system libraries, so make test-e2e-browser fails on a machine where that has not been done. The shell checks (make test-e2e) need only a running stack.

make build is not in any gate. It builds the three images from infra/Dockerfile.* — which do exist, contrary to what this section said for a while — but nothing runs it on a change, so it is the target most likely to have broken since anyone last tried.