How we build reliable Go software: the process and its evidence

nfltr is a relay, a stream multiplexer under it (link), agents and sidecars on other people's machines, and a hub that runs Claude Code agents across a fleet, all in Go, on networks we do not control. Failures there rarely leave a stack trace: they show up as a truncated download, a session that never ends, or a test that fails once in fifty runs on someone else's laptop.

This post describes the process we follow and the evidence it recorded, failures included. Most of the evidence comes from one laptop, Docker and local Kubernetes (kind) clusters; production is one VM. What we have and have not shown about scale says where that evidence stops.

What the process found

  • Concurrency bugs the race detector cannot see: a deadlock and a close that overtook data (38 of 100 responses truncated).
  • Tests that waited for time: five intermittent failures made reproducible by starving the machine; later rounds found a product bug and a gap that is still open.
  • Linearizability violations in every task store, four root causes, each pinned by a test that fails on the old code.
  • Seven relay bugs from other people's test suites, one a crash over HTTP/2 that needed no login.
  • No data error in any injected network fault we ran, and one scenario class that failed until the relay stopped keeping shares only in memory.
  • Three Linux-only product bugs found on the first day of a Linux runner.

Contents: layers · guarantees · design rules · repositories · testing · fault injection · scale · static analysis · performance · production signals · CI and releases · Claude Code · process · limits · principles · fit

The layers and what each guarantees

Each layer owns a few guarantees, and each guarantee has a technique that checks it. The rest of the post is organised around that map.

nfltr's layers, what each guarantees and what checks it Four stacked layers. The hub owns task ownership, leases, fenced writes and exactly-once completion delivery, checked by porcupine, a history oracle, fake-CLI end-to-end runs and fault soaks. The relay owns routing, sessions, drain and restarts and the refusal of old clients, checked by data-path tests, conformance suites, network faults and kind soaks. link owns multiplexing, per-session flow control and liveness, checked by linkvet, linkdebug, benchmarks with guard tests and a fault rig. Agents and sidecars own reconnects, local sockets and deadlines carried end to end, checked by network faults, gRPC liveness tests, test-under-load and a hermetic test guard. hub orchestration Owns: task ownership, leases, fencing, exactly-once completion delivery Checked by: porcupine, history oracle, fake-CLI end-to-end runs, fault soaks relay routing, sessions Owns: routing, sessions, drain, restarts, refusal of old clients Checked by: data-path tests, conformance suites, network faults, kind soaks link multiplexer Owns: multiplexing, per-session credit, liveness: a dead peer within a bound Checked by: linkvet, linkdebug, benchmarks with guard tests, fault rig agents and sidecars Owns: reconnects, local sockets, deadlines carried end to end Checked by: network faults, gRPC liveness, test-under-load, test guard
Each layer runs over the one below it. The hub's agents reach their nodes through the relay; the relay and the agents talk over link. Each layer is also its own module or package tree (see repositories).

What this means if you build on nfltr

What a user can rely on, how we check it, and under which conditions it was measured:

guarantee technique evidence
A dead path is noticed within a bound link pings each connection once it hears nothing 15.0 s with the default preset (3 of 3 runs), 4.5 s with fast, loopback TLS (link post); an idle gRPC stream through the sidecar ended in 14.9 s (gRPC post)
No reordered, duplicated or corrupted bytes under the faults we ran sequence numbers and CRCs checked at both ends under injected faults 0 data errors in 15 scenarios (network faults); not a general guarantee for faults we did not run
One slow reader holds only its own session per-session credit in link 8 of 8 other downloads finished beside a stuck reader, against 0 of 8 before
Bounded memory per session byte-bounded queues and per-session credit queue bounds, not total session memory: see what the bound counts
An idle agent is cheap idle budgets in tests 9.0–9.1 goroutines, 1.3 file descriptors per idle agent on the relay; test budget 11 goroutines and 132 KiB
A relay restart costs a retry, not data drain, persisted shares, reconnect on readiness kill -9: longest outage 2.9–3.2 s, 0 data errors
Old clients are refused with a reason a client generation checked at connect 20 old agents retrying for 60 s: 1,355 refusals, one log line, 0.76 s of relay CPU; the client is told to update
A finished agent's result arrives once durable hub inbox, reattach 50 of 50 disconnect-and-reattach runs delivered

Design rules

Fail closed at typed boundaries. Input that crosses a process or trust boundary is decoded once, at the edge, into a typed value and validated there. A malformed value is rejected with a structured error the caller can act on, never coerced, and prose is never scraped to make a decision. An agent's result, for example, is checked against the contract the hub declared (require_object, allowed_statuses, required_fields); one that does not match fails the turn.

Side effects at the edges. Validation, planning and rendering are deterministic functions; filesystems, subprocesses, networks and clocks sit behind small adapters. That lets most tests run without infrastructure and lets a model checker drive a store with a fake clock (linearizability).

Idle means near dormant. Periodic work first checks a constant-time change signal and does nothing when it has not moved; caches expire lazily instead of running janitor goroutines.

Evidence: an idle relay

Setup
Relay server releases of 2026-10-16 and 2026-10-17, idle with no agents connected; counts from the process itself.
Numbers
Idle wakeups about 1,430 per 65 s before, only the runtime's after; idle periodic goroutines 18 → 1 → 0.
Reproduce
/debug/pprof/goroutine?debug=2 of an idle process over time; a test asserts the count.
Limits
One machine, no agents connected.

Wait for events, not timers. A poll interval is a latency floor and a CPU cost. The relay now holds an open to a connecting agent until its stream registers, instead of the sidecar asking every 250 ms; a reconnect ends its wait when the channel is ready.

Evidence: waiting for the event instead of a timer

Setup
Sidecar and relay releases of 2026-10-26 and 2026-10-27, Docker on one host; a sidecar's uplink dropped silently by a firewall rule; an agent connecting while a client opens a session to it.
Numbers
Agent up → held open answered: 197–201 ms → 1 ms; opens while held 4/s → 2/s. Sidecar back after a 180 s silent drop: 2.90 s → 0.37–0.60 s. Idle cost unchanged.
Reproduce
Drop a host's traffic with iptables for 60–180 s and time the first successful call after.
Limits
Single-host Docker; a firewall rule, not a real outage.

Bound memory per session in bytes. Every queue between a reader and a writer has a byte bound, so a slow reader holds only its own memory. What the bounds count, and what they do not:

  • Relay TCP path: a hard bound on queued data of 1 MiB per session from agent to client, 64 frames of up to 64 KiB per agent from client to agent, and a 64 KiB read buffer on each side. gRPC and TLS buffers and per-session bookkeeping are not inside these bounds; the recorded measure of the whole is peak live heap of relay and agent together (8 sessions × 8 MiB upload: 51–70 MiB) and, from a later bench, about 40 KiB of heap per held idle TCP session.
  • link: a session's receive window (1 MiB by default) bounds the payload bytes it holds unacknowledged; credit is charged before a payload is queued. The queue's memory for those bytes is at most about 2× the bytes, plus one 4 KiB chunk for a session with a few bytes waiting. The connection's read buffer (2 KiB idle, a pooled 64 KiB while streaming), TLS buffers and session metadata are outside it; a total per session is not recorded.

Repositories and modules

The layers above are also separate Go modules, in one direction: link (standard library only) → relay proto → relay client SDK → relay core → relay server → nfltr. Each requires the ones before it and never the ones after; each has its own tests, fuzz targets and CI command. link, the relay SDK and protocol, and the relay core are public, and each public repository runs GitHub Actions on push and pull request with public dependencies only. Inside nfltr, code is layered domain → application → adapters → platform; transport types stay inside their adapter, with import-boundary tests; an ownership map keeps parallel lanes out of the same files; and a size budget stops a file over 1,000 lines from growing. Dependencies are vendored, so builds and tests run offline, and the relay binary is built from the tagged module nfltr requires. The three shipped binaries are their own boundary: a link-table test keeps the client binary free of orchestration, MCP and relay server code, which is where much of the size and start-up savings came from.

Splitting is reversible. Six orchestration contract modules were split out and folded back on 2026-10-09, because only this repository used them; a package now becomes a module only when it imports nothing internal, its tests pass alone, and an outside consumer would use it.

The cost is version skew. Under Go's minimum version selection, a module that requires an older link still builds with the newer one the final build selects, so its own tests never ran on what ships: the relay client SDK stayed on link v0.4.0 while nfltr built agents with v0.4.1. A check now fails unless every one of our modules in the build requires the selected link, and a change at the bottom is promoted in one round: each repository is bumped, the untagged versions are seeded where the CI runners can fetch them, the Linux runner checks every head at once, and tags are cut top-down only when all are green.

Evidence: ordered promotion, including the runs that failed

Setup
The promotion tool against the six repositories and the Linux runner, 2026-10-30, plus its own test with local repositories and a stand-in runner.
Numbers
One round instead of one per hop: about 16 min of runner time, against five sequential rounds and about 30 min. Failures the same day: a run tagged the relay server, then nfltr's push was refused because another change landed on main during the round (the tool now rebases and runs another round before any tag); a relay-server-only plan tagged and then failed nfltr's requirement check; a hand-written one-round plan left out link's nested QUIC module, so the requirement check failed after the tags were pushed and a follow-up bump was needed; and the relay repository's CI needs its own workspace file, which the tool had turned off for every command.
Reproduce
Bump, run every repository's CI against the untagged versions, tag only when all are green, then check with go list -m all that every module requires what the final build selects.
Limits
The workspace and main-moved fixes have tests in the tool's own test; the plan that left out the QUIC module was fixed by a note in the plan file, not a test. One more cost: git tests run from a git worktree once rewrote the main checkout's config, so work that runs tests uses separate clones.

Testing

Behaviour changes start with a test that fails on the old code; a test that guards a property must reject a deliberately broken implementation. Five of the examples below follow one pattern: the problem, why existing checks missed it, the reproduction, the root cause, the change, the guard, and what is still open.

Example 1: the race detector is not enough

The relay repositories run every package under -race in CI; in the CLI repository -race runs on the packages a change touches and in full before a merge, and a flaky test gets go test -race -count=50 before anyone believes a fix.

  • Problem. Downloads through the relay's TCP path were truncated: the field saw 32 % of connection-per-request vhost requests fail with unexpected EOF.
  • Why checks missed it. -race reported nothing: both were ordering bugs, not data races. Keep-alive traffic hid the truncation, because the connection stays open.
  • Reproduction. A data-path test: eight sessions echoing, uploading and downloading on one agent, and a target that writes then closes, checked byte for byte. 38 of 100 requests failed; the eight-session test deadlocked.
  • Root cause. Every session reader called stream.Send on one gRPC stream concurrently, which grpc-go forbids; readers parked in grpc's write quota while both receive loops sat idle. Separately, the relay selected between a session's data channel and its closed channel, and select often picked the close before the last data.
  • Change. One sender goroutine per stream draining a 64-frame queue, and a per-session byte queue that delivers everything queued before a close.
  • Guard. The data-path tests and a throughput benchmark with peak heap, run before and after every data-path change.
  • Limits. Upload with eight sessions got slower, in exchange for half the memory.

Evidence: the relay's TCP data path before and after

Setup
An in-process relay and agent over loopback TLS gRPC with the production transport options, one laptop, 2026-10-06; 1 and 8 sessions, plus a 40 ms RTT case.
Numbers
Connection-per-request requests failing before: 38 of 100 (an after count is not recorded; the after rows below completed). Download, 1 session: truncated → 720–757 MB/s; 8 sessions: deadlock → 1,195–1,387 MB/s. Upload, 8 sessions: 1,799–1,821 → 1,432–1,590 MB/s (noisy), peak heap 139–148 → 64–70 MiB. At 40 ms RTT, upload 72 → 254–258 MB/s.
Reproduce
In the public github.com/nfltr/relay-core: go test -race -run TestTCPTunnel ./core/session/ and -bench BenchmarkTCPTunnelThroughput.
Limits
Loopback on one machine, ranges of a few runs, no benchstat.

Hermetic tests

A test must not reach the network, a real relay or a real agent CLI. A guard installed from TestMain puts failing stubs for every agent CLI first on PATH (the binary fails if one ran), points relay addresses at a loopback trap, moves HOME to a temporary directory, and tells every child the test binary's PID so it exits after even a kill -9; a static check fails a package that needs the guard and lacks it.

Two rules came from flakes: fake executables run once before use (macOS assesses a new executable on its first run, 0.2–0.9 s idle, longer than a 3 s product probe under load), and servers under test listen on port 0 and report the port they bound (a helper that returned a closed :0 port handed two callers the same one).

Example 2: tests that waited for time

  • Problem. Tests that passed on an idle laptop failed now and then on a loaded one, and in CI.
  • Why checks missed it. At normal priority, even under heavy load, they passed: the failures needed a starved process.
  • Reproduction. Build the test binaries, then run them with GOMAXPROCS reduced at background priority while an uncached build loops beside them:
go test -c -o pkg.test ./pkg/
( while :; do go build -a -o /dev/null ./cmd/... ; done ) & load=$!
GOMAXPROCS=2 nice -n 19 ./pkg.test -test.count=5 -test.timeout=90m; status=$?
kill $load; exit $status
  • Root cause. In the first round, all five failing tests waited a fixed time for something a starved machine does later: an agent released 1.5 s after being asked to move, 50 visits wanted within 3 s, a SIGTERM sent 80 ms after start that killed the test binary itself because the handler was not installed yet.
  • Change. A test waits for the event or state it asserts; a wall-clock bound only catches a hang:
// Before: the backend sent the body 2 s after the headers, and the test
// wanted the headers within 1 s. A starved relay path took 1.1-4.4 s.
gotHeaders := make(chan struct{})
backend := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	w.WriteHeader(http.StatusOK)
	w.(http.Flusher).Flush()
	<-gotHeaders // the body follows the event, not time.Sleep
	io.WriteString(w, "body")
})
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) // hang bound only
defer cancel()
resp, err := client.Do(req.WithContext(ctx)) // returns once headers arrive
if err != nil {
	t.Fatalf("no headers within 20s (held until the body?): %v", err)
}
close(gotHeaders)
  • Guard. Each fixed test fails against a mutant (a backend that never flushes its headers: "no headers within 20s").
  • Limits. Later rounds were not only the tests' fault: a start-up probe that rode out a refused connection but not an HTTP 429 (fixed, with a test red before), and a memory test that read heap in use, which load distorts (it now reads the median of seven live-heap readings). One product gap is open: on the agent's link path, a backend reading slower than about 270 KB/s after the client's last byte can still get a 504, because the relay cannot see the agent's receive window drain.

Evidence: tests under load

Setup
A 10-core Mac at load average 19–33 with other work running; GOMAXPROCS 1 or 2, background priority, go build -a looping beside; 2026-10-29 and 2026-10-30.
Numbers
Before: drain-move test red 2 of 3 runs, headers-before-body red 3 of 3, one package killed by its own SIGTERM. After: all five green 5 of 5. Large share upload 2 of 12 → 0 of 20; large share download 7 of 12 → 0 of 12.
Reproduce
The shell loop above against your own packages, at normal and at background priority.
Limits
One machine, a handful of packages. The full CLI suite starved is too slow for a nightly run as it stands, so this is not yet a nightly gate.

Example 3: linearizability of the task stores

  • Problem. The hub's task stores (in memory, SQLite, Redis) implement fenced ownership: replicas take leases and acquire tasks, two owners must never both write, and a stale epoch must never win.
  • Why checks missed it. The conformance suite proved single-threaded behaviour; the bugs needed concurrent histories with lease expiry at a chosen moment.
  • Reproduction. porcupine checks recorded histories against a model written in Go, inside go test. Several replica goroutines run seeded operations against each real store while a test clock expires leases on purpose, with renewal pauses, crashes and dropped Redis connections (a transport error is an unknown outcome). The model is written from the contract, not the stores' helpers, and rejects hand-written illegal histories and a store that ignores fences. (Jepsen was rejected: it needs a JVM and SSH-managed nodes.)
  • Root cause. Three violations shared one cause: an acquire judged the caller's lease and the owner's lease at two different instants. The fourth was a lost reply: go-redis retried a renew the server had applied, extending the lease twice.
  • Change. One clock read per operation, taken under the write lock; Redis watches both lease keys; renew is a Lua script keyed by a per-call token.
  • Guard. One test per root cause, each red on the old store; an 8-seed slice in every go test (about 2.5 s).
  • Limits. Fenced writes never broke; the defects let a replica believe it held a task it did not.

Evidence: linearizability of the task stores

Setup
porcupine v1.3.1 in test files only; in-memory, SQLite and Redis (miniredis) stores; 2026-09-25.
Numbers
First runs (100 seeds × 200 operations per replica): violations in every store. After the fixes: 500 seeds × 3 replicas × 200 operations per store, about 630k checked operations (1.2k with unknown outcome), linearizable for all four targets in about 3 minutes under -race.
Reproduce
porcupine's README shows a key-value model; ours is one partition per task with owner, epoch and terminal state plus lease expiries.
Limits
Redis is miniredis, in process; no real network between replicas.

Conformance against other people's suites

Third-party suites encode someone else's reading of a protocol. We run them against a local relay, agent and client: Autobahn, h2spec, request-smuggling tools, testssl.sh, grpc-go interop, ssh-audit, kernel WireGuard and wireguard-go, Schemathesis, an OWASP ZAP passive scan, metrics linting and an MCP protocol client. A process that dies during a suite fails the run.

Evidence: conformance suites

Setup
A local relay, agent and client on an internal Docker network, 2026-10-07, plus a read-only TLS scan of the public endpoints; further rounds on 2026-10-08.
Numbers
Autobahn: 112 and 107 failures of 247 before, 0 after; smuggling: 134 of 134 mutations handled; gRPC interop: 28 of 28; seven bugs fixed in the first round, including a relay crash over HTTP/2 that needed no login.
Reproduce
testssl.sh against the public endpoints, which any visitor can run; the other suites are open source and run against any HTTP/2, WebSocket or gRPC server.
Limits
One pass per suite; protocol behaviour, not load.

Four rounds of these suites (the protocols, then SSH, WireGuard and STUN, then metrics, web and MCP surfaces, then static analysis and image scans) fixed 21 bugs and left 18 findings open when their reports were written. We classify all 18 below as security vulnerability, hardening, conformance deviation or compatibility, from the reports' own triage where they give one and ours otherwise; rows group related findings, and details that would help an attacker are left out.

class finding (suite) status
security vulnerability an authenticated peer could make the other side grow memory through flow control (fuzzing) fixed since
security vulnerability one CLI command ignored the saved certificate pin (client TLS matrix) fixed since
security vulnerability, low an old vulnerable library reached through a message-broker client (govulncheck) fixed since: the dependency is gone
hardening no post-quantum key exchange on the end-to-end TLS endpoint (testssl.sh); one CLI command had no pin option (client TLS matrix) both fixed since
hardening no SSH connection throttling (ssh-audit); the web content policy allows inline scripts (OWASP ZAP); a weak RSA key from a private CA accepted on one platform (client TLS matrix); a medium base-image library issue with no patched package yet (trivy) open: out of scope, documented, a policy decision, awaiting a patch
hardening hub metrics grew with every worker seen; an unused helper and a lookup table that only grows; TURN usernames visible to the TURN operator (promtool, STUN/TURN suite) hub metrics fixed since; the rest open
conformance deviation the API description differs from some real responses; unknown API paths answer a misleading 404 (Schemathesis); latency metrics in milliseconds, not seconds (promtool) open; one metric family retired
compatibility SSH key exchanges are not post-quantum, kept for older clients (ssh-audit); the MCP server answers with an older protocol version, which the spec allows (MCP Inspector); a UDP mapping reused for TCP (STUN/TURN suite) open, by choice

About 750 lower-risk static-analysis hits were read and put in a baseline; since then a release cannot ship over a new untriaged finding.

Every wire codec, the SNI peek, share codes and mailbox ingress also have Go fuzz targets in the repositories that own them; a crasher is committed with its fix as a regression test.

History oracles and runs with no model calls

An offline oracle judges the hub from its recorded task events: exactly-once acceptance, an ordered cursor per task, no rejected output integrated, a maximum silence. End-to-end tests use a scripted stand-in for the Claude CLI, checked by a parity test against our real parsers.

Evidence: hub primitives, 50 runs each

Setup
One local relay (SQLite store), workers, a joined node and one hub; zero model calls; a shared laptop (load1 up to 9.6); 2026-09-27.
Numbers
50 of 50 each for spawn (by machine, by workspace, with a repository clone), steer, continue, stop, budget stop, disconnect plus reattach, and no orphan processes or worktrees. Start p50 60–267 ms, delivery p50 36–382 ms by primitive.
Reproduce
A fake CLI replaying recorded streams, plus a parity test.
Limits
Rates on one host, not gates. The fake proves the machinery, not model quality.

Fault injection

Example 4: network faults, judged by the data

A Docker Compose harness runs a relay, an agent and a client with a connect sidecar, and three channels: request/response (rr) and a server stream through sidecar mappings, and HTTP through a share. Faults come from iptables and tc netem per container; a fault proxy would change the TCP behaviour being measured.

What "0 data errors" means here. Every request/response frame carries a sequence number, random 64 B–64 KiB of payload and a CRC32; the reply must carry the request's sequence and bytes, and the server must never see a sequence twice or out of order within a connection. Stream frames must arrive 0, 1, 2, ... with intact payloads; HTTP responses must carry the requested sequence and exactly the expected bytes. Both ends check. The claim covers these 15 scenarios on one host, nothing else.

What it does not mean. It is not delivery across an interruption. A request in flight when a path breaks fails, and each request has a 10 s deadline. The test application, not nfltr, recovers: it closes the broken connection, opens a new one 250 ms later, and counts every failed attempt as a failed operation, never as a data error. nfltr does not buffer a connection's data across a disconnect; the harness does not test that. Order and gap checks run within one connection (a stream starts again at frame 0 on a new one); the server's replay check spans connections, so a request that arrived twice after a reconnect would count as a data error. Whether a retried request reuses its sequence number is not recorded.

  • Problem. After a relay restart, kill -9 or a reconnect storm, the HTTP channel never came back.
  • Why checks missed it. Unit and soak tests restarted relays, but no test kept an HTTP share open across a restart.
  • Reproduction. The harness's relay-restart, relay-kill9 and storm scenarios failed every time; the data was never wrong.
  • Root cause. The relay kept HTTP shares only in memory; after a restart the share URL answered 401 until the agent restarted.
  • Change. Shares are persisted on the relay.
  • Guard. The same scenarios now pass without the driver's repair step, in the nightly subset.
  • Limits. Single-host Docker, not a WAN. NAT rebinding over TCP costs one liveness bound (about 15 s); the QUIC agent link is measured separately.
scenario failed ops rr / stream / http longest outage, rr / stream / http (s) first success after heal (s)
partition-agent-120s 333 / 331 / 321 120.8 / 120.7 / 120.9 0.5 / 0.7 / 0.6
latency +200 ms 0 / 0 / 0 1.2 / 0.5 / 2.1 0.4 / 0.1 / 0.4
loss 20 % 4 / 3 / 2 45.5 / 34.6 / 13.0 1.9 / 2.1 / 0.2
nat-rebind 2 / 2 / 2 15.5 / 15.5 / 15.6 15.2 / 15.3 / 15.5
relay kill -9 (after the fix) 11 / 11 / 11 2.9 / 3.2 / 3.0 0.3 / 0.5 / 0.3
storm, 200 agents (after the fix) 5 / 6 / 6 1.4 / 1.9 / 1.7 0.2 / 0.6 / 0.4

Evidence: network faults

Setup
One Docker host (10 CPUs) shared with a 24 h kind soak, load average 19–33 at the start; relay built from the tree under test; 2026-10-26, the fixed scenarios rerun later at load 27–50.
Numbers
0 data errors in all 15 scenarios. 1 % and 5 % loss and +200 ms RTT: no failed operation. 200 agents all back 3.8 s after a relay restart (3.0 s after the fix), relay peak 12 % of a core and 77 MB RSS.
Reproduce
Any Docker host with NET_ADMIN and sch_netem: per-container iptables -j DROP and tc qdisc add dev eth0 root netem delay 100ms 20ms, with sequence-numbered, checksummed traffic.
Limits
One run per scenario on one loaded host; no real WAN.

The gRPC half (keepalives, deadlines, stuck handlers) has its own post. For the protocol cut, we also ran released old clients against the new relay: 20 old agents relaunched every second for 60 s made 1,355 refused connections, one log line, and cost the relay 0.76 s of CPU (1.27 % of a core).

What we have and have not shown about scale

The relay can run as several replicas behind one Service; the scale-out post covers the design. All of that evidence is local.

Measured. Soaks of three replicas on three kind nodes on one 10-CPU laptop, with Redis and Postgres in the cluster, rotate faults every 20 minutes: rollout, pod kill, an empty Redis restart, a 30 s one-replica partition and, with Chaos Mesh, clock skew. A judge checks fixed bounds and per-pod goroutines, file descriptors and memory.

  • The first runs failed. 41 % of cross-replica TCP visits failed once an inventory record, written once with a TTL, lapsed. The first 6-hour run failed two bounds (a 62 s partition recovery against 50 s, pod kills near 40 s). A 24-hour run two weeks later missed four bounds; each cause got a guard test. A short soak after those fixes still missed two rollout bounds, for other causes; they are open.
  • What "99.999 %" means. The second 6-hour run (2026-10-10), 52 agents (40 HTTP, 12 TCP) in one process, one visit per agent per second, each visit judged on its own. Each visit is one attempt: it fails unless it returns HTTP 200 with the expected body within a 10 s client timeout, and nothing is retried. Of 858,515 HTTP and 257,256 TCP visits, 99.696 % and 99.860 % succeeded; that figure includes the faults. Steady availability excludes visits that ended inside a fault window or within the 50 s recovery allowed after it: 816,632 and 245,020 steady visits, 99.9990 % and 99.9992 % successful, which the report gives as percentages, not counts (about 8 and 2 failed visits). A 6-hour soak on one laptop is not annual five nines, which allows about 5 minutes of downtime in a year.
  • What "0 s pod-kill recovery" means. The time from the end of the fault to the last failed visit after it: no visit failed after any of the four pod kills ended. During each kill, 14–36 visits failed.
  • Right-sizing. One replica's knee is about 1,300 visits/s at 1.3 cores; under soak load each agent adds 2.4–2.8 mCPU, 0.9–1.5 MiB and 14–16 goroutines; each held TCP session adds about 40 KiB of heap (6,000 sessions held). An autoscaler run reached 3 replicas in 36 s and the rebalancer split agents 33/33/33, but CPU did not spread evenly (1.05/1.02/0.48 cores): 10 of 12 criteria passed.
  • Largest counts. 500 agents registering in process on one machine; a storm of 200 agents reconnecting. A 7-hour soak on a managed cloud cluster (3–5 replicas of half a core) reached 99.82–99.90 % steady availability, and 21 of 26 recoveries exceeded the 50 s bound.

Not shown. Production is one VM, and the chart has carried no production traffic. A move to a cluster is triggered by measured alerts (relay CPU above 0.25 cores sustained, resident memory above 512 MiB, more than 250 connected agents); production is below all three.

experiment that would show it what it would measure status
Production on several replicas rollouts, pod loss and rebalancing under real traffic not yet run
Thousands of agents on a cluster per-pod limits, directory load, reconnect storms at that size not yet run
Replicas in different networks replica-link liveness and forwarding over real WAN latency and loss not yet run
Partitions between relays in production recovery bounds outside a lab not yet run
A capacity curve: 0–10k agents against 1, 2, 4 and 8 replicas throughput, memory, CPU, recovery and tail latency per point planned, not yet measured
The cloud soak meeting its bounds rollout and recovery bounds on slower, shared nodes run; bounds not met

Static analysis

  • Custom analyzers: link ships linkvet, go vet analyzers for link sessions and gRPC clients (a stream neither read to io.EOF nor cancelled, calls with no deadline, a connection per request). Its first run over our code found one leaked stream; a deliberately broken file proved zero findings meant something. link's linkdebug build tag reports sessions collected unclosed, and found a production bug: a forward path that did not close its session when the route failed late.
  • Scanners with baselines: gosec, semgrep, hadolint and trivy fail on a new finding or a stale baseline entry; govulncheck allow-list entries name their removal condition. go vet, deadcode on the shipped binaries, cross-builds for every shipped platform, and a bash 3.2 check for build scripts run too.

Performance

Every change that claims a gain captures profiles at its base commit and after: CPU, heap, mutex, block and goroutines, plus benchmark text from -count=6 or more for benchstat. On a shared machine, old and new run interleaved and we report medians; a change inside run-to-run noise is reported as none. Every gain adds a line to a ledger naming the guard test that keeps it, and a comment at the optimized code points to that test.

Both link numbers come from one 10-core Apple M-series laptop (macOS, arm64), in one process over loopback TCP with TLS off, measuring one-way throughput of one connection; the Go version is not recorded in the benchmark notes. In detail:

number benchmark transport and path machine and runs public
link, 64 B × 8 sessions: 56 → 335 MB/s, 0 allocations per message BenchmarkThroughput: one-way MB/s of one connection TCP over loopback, one process, TLS off (TLS 1.3 rows separately: small messages +27–84 %) a 10-core Apple M-series laptop shared with other work (load 8.6–11.4); 8 runs per side, old and new alternating, benchstat medians; deltas under 10 % read as noise yes, in github.com/nfltr/link
link, 64 streams × 64 KiB: 2.35 → 8.06 GiB/s the compare/defaults rig, link alone on one TCP connection one machine; the rig's README has the setup same laptop; 8 alternating passes, headline from the first 6 (load 5–8, p = 0.002); all 8 give 2.32 → 7.30 GiB/s; BenchmarkThroughput at load 10–73 gave 2.53 → 6.67 GiB/s yes, compare/ in the link repository
relay front middleware: 78 → 35 allocations, 3.95 → 2.44 µs per request a middleware benchmark, in process, no network none: handler chain only same laptop; allocations exact over 8 alternating runs; time from 10 runs per side at load 8.1–9.6, benchstat no
start-up allocations of our own packages: 16.2k / 1.30 MB → 2.7k / 0.23 MB GODEBUG=inittrace=1 nfltr version, summed over our packages none a Linux arm64 container; counts are deterministic per build; timings beside them from perf stat -r 100 no (the method works on any binary)
CLI size: 88.4 → 61.2 MB byte count of the build none stripped CGO_ENABLED=0 -ldflags "-s -w", linux/amd64, go1.26.6 no

Guard tests named in the ledger include TestSteadyStateAllocs and TestManyWritersWakeForTheirTurn (link) and TestSpawnAgentsSavesDoNotGrowWithTheAgents (hub group commit: 24 → about 4 state writes when spawning 8 agents). We compared link against yamux before keeping it; that comparison found three link defects, fixed in the next release (link post).

Start-up and size budgets

Go runs the initialisers of every linked package in every process, and our CLI starts once per agent. Package-level regular expressions were 13.4 % of nfltr version's CPU. What a command may never need is now built on first use:

// Before: compiled in every process, whatever the subcommand.
var tagRE = regexp.MustCompile(`^v(\d+)\.(\d+)\.(\d+)$`)

// After: compiled the first time a caller needs it.
var tagRE = sync.OnceValue(func() *regexp.Regexp {
	return regexp.MustCompile(`^v(\d+)\.(\d+)\.(\d+)$`)
})

For regular expressions a wrapper like the toolchain's cmd/internal/lazyregexp keeps call sites unchanged. A test runs GODEBUG=inittrace=1 nfltr version and fails above 3,000 allocations or 280 KiB in our own packages. Remaining start-up is mostly loading the system root certificates for the first TLS verification (36 %), which we keep. Each shipped binary also has a size budget (measured size plus 5 %); the rule is to cut the import edge that grew it before raising the budget. One embedded policy engine was 12.2 MB. Profiles also feed a PGO build of the hosted relay; we have not recorded a gain from it and claim none.

Detecting failures in production

What we run against production, and what it does not cover:

  • An hourly canary spawns agents on canary nodes through the production relay, checks exactly-once delivery, a pushed commit and cleanup, and records one line per run with a typed failure reason, spawn → first progress and spawn → completion (p50 and max). Its first 50 agents gave first progress p50 3.3 s, p95 5.7 s, max 6.6 s, and completion p50 8.5 s, p95 10.9 s, max 11.6 s, of which 3 s is the stand-in agent's own work. A run counts as degraded when latency exceeds both 1.5× and +2 s over a 7-day baseline.
  • The status page shows the canary's 24-hour and 7-day success rate with p50 and p95; it is one machine's view.
  • Relay metrics are scraped every 30 s through an owner-only agent: connected agents, a connected gauge per service the relay depends on, resident memory, and an alert when the p95 time to open a path stays above 2 s for 15 minutes.
  • Dead peers are found by link's liveness presets: 4.5 s (fast), 15 s (default), 30 s, 45 s and 60 s for the slower presets, measured on loopback. Each sidecar reports open and failed counts and its last error on /status.

Not measured yet: p99 latency in production; memory per session in production (only per process); latency across real networks between hosts. The reconnect times quoted in this post come from Docker on one host, not from production.

CI and releases

CI is one command per repository that reads its steps from the repository's workflow file and fails closed on syntax it does not know. GitHub Actions runs too, but it is not the gate: its results do not reach the release. The pre-push hook runs the command for pushes to main and tags, and is a convenience: it can be bypassed with a reason, which is logged. The enforcement is the release's gates step: a model-free Linux runner runs the same command on every new sha of every repository in a fresh clone, and the release refuses to ship unless every sha it ships has a green result there.

Example 5: what a Linux runner and re-runnable releases found

  • Problem. Code that passed on the Mac failed for Linux users, and three relay server tags had been cut on commits that turned out red.
  • Why checks missed it. The pre-push run happens on macOS; Linux-only behaviour (process dumpability, kernel queue sizes, scheduling) never shows there.
  • Root cause. Found in 60 runs on the runner's first day: three product bugs, one a key-holding process left dumpable on Linux, and ten test races.
  • Change. All fixed the same day without sleeps; red to fix took 27–51 minutes for the two product bugs the runner caught. Tags now need a green Linux result first.
  • Limits. Relay server CI takes 437 s (from 494 s), not the 4 minutes we aimed for: -race is CPU-bound on the runner's four CPUs.

Releases run as steps that each print one JSON line (ok, exit, a reason, facts); a Claude Code hub runs them on a build node and judges each step only by that line (nfltr on nfltr). Production steps run only with a grant the owner signed for that exact step, one turn per grant. Making each step safe to re-run found bugs before any release used it: re-deploying the live version made the rollback target the same release, rolling back twice rolled forward again (for the relay and for sites), and re-deploying a site three times evicted the release a rollback needed.

Building with Claude Code: what changed

Most of this work is done by Claude Code agents: 1,277 of the 1,320 commits in the nfltr repository since 1 October carry a Claude co-author line. The work is split into lanes. A coordinating session plans, gives each lane a scoped brief (the files it owns, the tests it must pass, what it must not touch), integrates the results and runs the final verification. Each lane is an agent session in its own clone, at most about three heavy lanes per machine; some run as agents on other machines through nfltr's own hub. Since lanes started recording wall time, 106 finished lanes took a median of about an hour.

What it made possible. A lane is only as good as its brief, so briefs are written as checks: a failing test, a measured target, a golden file. That made breadth affordable. link went from a decision document on 10 October to a replica link proven under a fault matrix on 11 October, its wire pinned by golden bytes, and a comparison against yamux the same day found three link defects, fixed in the next release. The flake lanes above ran beside releases, not instead of them. This post went through two outside review rounds; corrections were checked against the source documents, and some claims were narrowed because the documents did not support them.

Why the process got stricter. Agents make the same mistakes people do, faster and in parallel, so a rule that is only advice gets broken by some lane. The recorded incidents are agent incidents: a pattern-based pkill that took down other lanes' tests, git tests run from a worktree that rewrote the main checkout's config, a session stopped at its spending limit that lost unpushed work (two lanes were redone), parallel lanes tagging off main and on commits the Linux runner later marked red, a gate judged through a pipe, which let a failing test reach main. Each became an executable check or a one-line rule in the instructions file every agent reads at the start of a session; after each task, a task-analysis step turns its problems into edits of that file and deletes rules that went stale.

Guardrails that exist because agents run things.

  • Tests never run a real agent CLI: the hermetic guard puts failing stubs first on PATH, traps relay addresses and ties every child process to the test binary.
  • End-to-end runs use a scripted stand-in for the Claude CLI, so they make no model calls.
  • Agents are judged by structured results, not prose: the hub declares the fields an agent's result must have, and a release step is one JSON line.
  • Agents cannot release on their own: production steps run only with a grant the owner signed for that exact step, one turn per grant, enforced by the build node.
  • No concrete model or effort is a default anywhere in the code; a pre-commit check fails on one.

Limits and costs. The checks in this post exist because agent-written code needed them: the tests that waited a fixed time were written in this process, though who wrote each is not recorded. The pre-push run can be bypassed, with a logged reason. Review is still needed. Model spend is real; the operator sets caps per lane, and amounts are not given here. Some decisions stay with people: signing production grants, making a repository public, and what to build next.

Process

Every defect is fixed for the project, not the occurrence: name the failed invariant, fix the widest owning layer, and leave a check that fails if it returns. The rule does not enforce itself: the first fix for the dumpable process covered one binary, and a review found the same class elsewhere. Every architecture decision is a dated document with what was rejected and why.

The incidents above each left a rule: real exit statuses (set -o pipefail), separate clones, checkpoint branches, tags only on green commits on main, and stopping only a lane's own processes.

What does not work yet

  • Under-load testing is not a nightly gate for the full CLI suite.
  • One upload path can still time out when a backend reads slower than about 270 KB/s after the client finishes.
  • Two rollout bounds of the kind soaks are still missed, and the cloud soak has not met its bounds.
  • Scale beyond one VM in production is not shown.

The principles behind it

The process exists so that people who build on nfltr can predict how it behaves when things go wrong:

  • Make invalid states hard to represent: typed boundaries, fenced writes, one owner per decision.
  • Make failure detectable: liveness bounds, structured errors, typed refusals, a canary that says what failed.
  • Make correctness measurable: checksummed data under faults, linearizability, other people's suites, oracles over histories.
  • Keep performance sustainable with budgets: idle, start-up, size and allocation budgets, each with a guard test.
  • Keep operations predictable: green commits only, re-runnable release steps, owner-signed production steps, rollbacks that change nothing when repeated.

Is it for you?

Fits if you write Go services for networks or machines you do not control, and flaky tests or slow starts have been "fixed" with longer sleeps. Nothing here needs nfltr: a starved test run, porcupine, third-party suites, tc netem, benchstat and inittrace work in any Go repository.

Not ready for a copy-and-paste toolkit, or for evidence of scale beyond one host. Our harnesses live in the closed nfltr repository; this post describes their shape. link with linkvet, the relay SDK and the relay core are public (open-source post).

Try next: run your slowest-to-fail package with GOMAXPROCS=2, background priority and a build loop beside it, and see which waits break. Then read the relay comparison for the performance work and the scale-out post for the soaks.

← All posts