Developer guide

For people changing Loomscope's code. CONTRIBUTING.md covers setup and the pull-request rules; this guide covers how the system is put together and how to extend it.

Contents

• Architecture
• Repository layout
• The request path
• Tenant isolation
• The contract pipeline
• Adding a service signature
• Adding a scanner
• Adding a topology view
• Background jobs
• Realtime
• Database conventions
• Testing
• Known rough edges

---

Architecture

Loomscope is a control plane plus scanners. Its shape follows from three decisions worth reading before changing anything structural:

• ADR-0001 — REST and SSE rather than gRPC, because daemons live behind networks we do not control.
• ADR-0002 — PostgreSQL is the only datastore. No Redis, no broker.
• ADR-0007 — where the UI styling is going, and why it is currently inconsistent.

The daemon never accepts inbound connections in the default mode. It polls for jobs and posts observations, so it works from a branch office behind NAT with no firewall changes.

Repository layout

apps/server            Control plane — Next.js 15 App Router
  app/(app)/           Authenticated UI pages
  app/api/v1/          REST API: daemon endpoints and admin routes
  server/db/           Drizzle schema, migrations, scoped transactions
  server/topology/     The four topology generators, layout, persistence
  server/cloud/        AWS and Kubernetes collectors
  server/cve/          CVE matching
  server/snapshot/     Capture, diff, PDF attestation
  server/ai/           Assistant provider abstraction and tools
  server/auth/         Session, org scope, per-site permissions
apps/docs              Nextra documentation site
services/daemon        Go scanner
  internal/scan/       ICMP, ARP, TCP, UDP, DNS, TLS, SNMP, Docker
  internal/signatures/ Pattern engine, built-in signatures, YAML loader
  internal/netflow/    NetFlow v5/v9, IPFIX, sFlow parsers
services/cve-ingest    Optional standalone OSV/NVD mirror worker
packages/contracts     Zod schemas → OpenAPI → generated Go client
packages/shared        Shared TypeScript types
infra                  Compose files, Dockerfiles, systemd units

The request path

A discovery observation travels like this:

1. The daemon polls /api/v1/daemon/jobs/poll and receives a scan job.
2. It scans, batching results in internal/discovery/batcher.go.
3. It posts to /api/v1/daemon/observations.
4. apps/server/app/api/v1/daemon/observations/route.ts authenticates the daemon, opens an org-scoped transaction, and upserts hosts, ports, services, interfaces, certificates.
5. The upsert emits NOTIFY, which the SSE endpoint relays to open browsers.

That observations route is the hot path and the most consequential file in the control plane. It is also, currently, untested — see Known rough edges.

Tenant isolation

This is the part to understand before writing any query.

Every business table carries organization_id and has row-level security enabled and FORCEd. 0001_init.sql applies this in a loop over every table with that column, so new tables are covered automatically by having the column.

Access goes through a scoped transaction:

await withSessionOrg(async (tx, orgId) => {
  // SET LOCAL app.org_id has already run on this transaction.
  // RLS filters everything, even if this query forgets its WHERE clause.
});

Three wrappers exist:

Two tables reach their tenant through a parent rather than a column — membership_site_scopes and snapshot_entities. Both have an explicit policy that joins to the parent. If you add a child table without organization_id, you must write that policy; the loop will not cover you.

Never bypass this for an admin endpoint. Cross-org access requires an explicit superadmin scope and an audit entry.

The contract pipeline

packages/contracts holds Zod schemas and is the source of truth.

Zod schemas  →  OpenAPI 3.1  →  Go client
   (hand)       make openapi    make oapi-go

Changing a daemon-facing schema means: edit the Zod schema, run make openapi && make oapi-go, and commit the regenerated artefacts. A pre-commit hook does this via lint-staged; if you bypass hooks, do it by hand. A stale openapi.json means the published contract lies about the API.

Adding a service signature

Two paths, producing the same internal type.

Built-in, compiled into the daemon, for common services:

// services/daemon/internal/signatures/core/grafana.go
func (GrafanaSignature) ID() string { return "grafana" }
func (GrafanaSignature) DiscoveryPattern() pattern.Pattern {
    return pattern.Endpoint{
        Port:   pattern.PortHTTP,
        Path:   "/api/health",
        Expect: "grafana",
    }
}
func init() { registry.Register(GrafanaSignature{}) }

Custom, YAML loaded at boot and on SIGHUP, for anything in-house:

version: 1
service:
  id: internal-auth-service
  name: Internal Auth Service
  category: Web
  confidence_default: 85
  pattern:
    all_of:
      - port: { protocol: tcp, number: 8443 }
      - endpoint: { port: { protocol: tcp, number: 8443 }, tls: true,
                    path: /healthz, expect: "auth-service v" }

The pattern grammar is identical in both: port, endpoint, all_of, any_of, not, is_gateway, mac_vendor, subnet_type, none. The YAML loader validates strictly — an unknown field or bad enum fails at boot with a structured error rather than being ignored.

Prefer an endpoint match over a bare port. A port number is a guess; a response fragment is evidence, and the confidence score you assign propagates into CVE matching.

Adding a scanner

Scanners live in services/daemon/internal/scan/<name>/. The contract:

• Take ctx context.Context first, and honour cancellation. Scans get cancelled.
• Never assume IPv4. Use netip.Addr throughout and test both families.
• Emit through the batcher rather than posting directly — it handles batching and backpressure.
• Log with zerolog, structured fields, never string interpolation, never a credential.
• Provide a no-op implementation for platforms you do not support, behind a build tag, as arp_other.go does.

Then extend the observation contract in packages/contracts, regenerate, and handle the new fields in the observations route.

Adding a topology view

Generators live in server/topology/ and share a shape:

export async function generateX(tx: UiTx, siteId: string | null): Promise<TopologyPayload>

Return nodes and edges; persist.ts handles upserting, preserving user-moved positions, and layout. Two constraints that are easy to get wrong:

• groupId is a uuid foreign key to a parent node, not a label. It is cast to ::uuid on insert, so a prefixed string like site:abc throws. Non-structural grouping information belongs in node metadata.
• Layout engines differ: dagre for hierarchical views (L2, workloads), elk for graph-shaped ones (L3, application). elk ignores groupId entirely.

Background jobs

Schedulers start on the first request through the (app) layout, guarded by a singleton. They run in-process:

They use withForcedOrg because they have no session. In-process scheduling means multiple replicas would each run them — a real limitation, and the reason job claiming uses FOR UPDATE SKIP LOCKED.

Realtime

PostgreSQL NOTIFY → an SSE endpoint → the browser. Payloads are capped at 8000 bytes, so notifications carry identifiers and the client re-reads. Do not add websockets (ADR-0001).

Database conventions

• Migrations are SQL, forward-only, numbered, and never edited once shipped.
• SET lock_timeout = '3s' at the top of any migration that runs ALTER.
• Expand-and-contract for renames, type changes and new NOT NULL columns.
• Backfills batch 5000 rows, in their own migration.
• CREATE INDEX CONCURRENTLY is not available — the runner wraps each file in an implicit transaction. If a large live table needs an index, that migration has to be run outside the normal runner, deliberately.
• No raw SQL in business code without going through a scoped transaction.

squawk lints the migrations a change adds. Two rules are disabled for the reasons recorded in squawk.toml.

Testing

make typecheck     # tsc --noEmit across all packages
make test          # vitest + go test -race

What exists: 66 TypeScript tests over parsing, diffing, CVE matching, subnet derivation, permissions and export serialisation; Go tests over the pattern engine, SNMP table parsers, NetFlow parsers and the YAML loader; shell E2E scripts under scripts/ that run against a live stack.

Where to put a new test:

• Pure logic → a __tests__ directory next to the code.
• Anything touching the database → an integration test using the real PostgreSQL that CI provides. Do not mock Drizzle.
• Go → _test.go beside the file, with -race if goroutines are involved.

Known rough edges

Stated so you are not surprised, and because these are good first contributions.

No linter. The workspace has no ESLint configuration (ADR-0006). TypeScript strict mode and tsc carry the load.

58 unchecked casts. Raw SQL results are cast with as unknown as RowType, bypassing both Zod and the type checker. A wrong column name fails at runtime, not compile time.

Untested surfaces. All 19 API routes, including the observations hot path, and every UI page. There is no browser test framework.

Configuration is read at import time. Several modules throw on a missing DATABASE_URL when the module loads, so next build needs the variable set even though nothing connects. Making it lazy is on the roadmap.

The eBPF collector attaches no probes. It performs capability checks and returns a closed channel.

make build and make test-e2e do not work. The first builds a compose file with no build stanzas; the second invokes Playwright, which is not installed.

The UI styles colours inline. Around 307 inline colour strings against 12 Tailwind colour utilities, and dark mode is partly broken. See ADR-0007.