link: why relay replicas stopped talking gRPC

October 11, 2026

nfltr connects machines through a relay they dial out to: an agent on your machine opens the connection, and visitors reach its services through the relay, with no inbound port. Each agent, and each pair of relay replicas when the relay runs as several, carries hundreds of long-lived byte streams at once: TCP tunnels, end-to-end encrypted shares, proxied HTTP. Those streams rode on gRPC, and that cost us two things a tunnel cannot afford: a peer that vanished took 80 s to 2 h to be noticed, and one visitor who stopped reading could stall every other session on the same stream.

What we found: a stream multiplexer of our own, called link, removes both. A dead peer is found in 15 s by default (4.5 s with the fast preset), a stuck reader holds only its own session (8 of 8 other downloads finished, against 0 of 8), one forwarded session between replicas runs at 0.90 to 0.98 of same-replica throughput against 0.53 to 0.61, and about 4,000 lines of scaffolding went away. It is not better everywhere: eight parallel sessions still reach only 0.83 to 0.86, TCP head-of-line blocking under loss is untouched, and against yamux it is days old. Each claim below has its setup, a way to reproduce it and its limits.

What the move to link showed

Dead peers found in 15 s with the default preset (4.5 s with fast), against 80 s to 2 h over gRPC.
A stuck reader holds only its own session: 8 of 8 other downloads finished, against 0 of 8 cold over gRPC.
Forwarding between replicas got cheaper: one session at 0.90 to 0.98 of same-replica throughput, against 0.53 to 0.61; eight sessions only 0.83 to 0.86, below the 0.9 target.
Under 1 % loss, TCP still limits it: four connections were 3.5 to 5 times faster than one.
Against yamux: less memory per session and an error, not a clean EOF, on a cut connection; yamux found a dead peer sooner as configured, and the comparison found four defects in link, all fixed.
A 6-hour multi-replica soak passed, with one partition recovering in 26 s against a 10 s target.

The problem

Between two relay replicas, and between an agent and its relay, there are hundreds of sessions at once. Each is a byte stream with its own pace: a visitor downloading a large file, a terminal that sends a key now and then, a tunnel that sits idle for an hour. They all share one pair of endpoints, and the failure of one connection should cost its sessions an error, not minutes of silence.

Between two relay replicas: gRPC streams and their scaffolding before, a small pool of link connections after.

The question that started the work came from the Kubernetes soak: why does one IP-to-IP link between two pods need two different connection codes, one for agents and one for replicas? The answer was that it did not, and that the second one, built on gRPC, carried most of the problems below.

What gRPC between two pods gets wrong, and what link does

Liveness

gRPC keepalive clamps its ping interval to at least 10 s, counts only bytes received, and a write into a dead socket still succeeds. A pod that disappeared left visitors waiting for between 80 s (with the chart's settings) and 2 h (with the defaults). In link both ends ping once they have received nothing for the interval, every ping is answered, and the connection is dead within one interval plus one timeout of the peer's last frame, busy or idle. Open sessions then get an error. The default preset detects a dead peer in 15 s and the fast one in 4.5 s.

Evidence: a dead peer is found in 15 s, not 80 s to 2 h

Setup: Two relay replicas and an agent in one test process over loopback TLS, a visitor's session forwarded between the replicas, then the link cut; one Mac (Apple M5, 10 cores); 3 runs per preset; October 10 and 11, 2026. The gRPC figures are the times its keepalive settings give (the Helm chart's 60 s / 20 s and the code default of 2 h / 20 s), with the formula checked by one run scaled down to 10 s / 2 s, which took 12.0 s.
Numbers: gRPC: 80 s with the chart's settings, 2 h 20 s with the defaults. link: 15.0 s with default in 3 of 3 runs, 4.5 s with fast, at both ends.
Reproduce: (a) With the released binaries you choose the preset (nfltr http 3000 --keepalive fast) and can cut the network to watch the agent reconnect; no released tool times it. link's own bound is public: in github.com/nfltr/link, go test -run TestDeadPeer . (TestDeadPeerBusy, TestDeadPeerWorstCaseIdle). (b) Our check, in the closed relay-server repository: LINK_DEAD_PEER_PRESET=default go test -mod=vendor -run TestPeerLinkDeadPeer ./core/replicas/; and in the closed nfltr repository, with replicas in containers and real partitions: make test-replica-link-faults NEW_RELAY=<linux nfltr-relay> NFLTR=<linux nfltr> ($0, docker, about 25 minutes).
Limits: Three runs on loopback; the gRPC numbers are computed from its settings, not waited out for two hours. Before link v0.4.2 an idle connection cut right after a pong took up to 20 s against the 15 s bound (see the yamux comparison below).

Isolation and head-of-line blocking

Many sessions on one gRPC stream share one receive loop, so a stuck reader blocks the rest. In the test we call F1, with a cold start, 0 of 8 other downloads finished in 10 s while one reader was stuck. Giving each session its own gRPC stream fixes that, at +10.1 KiB and +4 goroutines per session. In link, each session has its own credit, nothing waits on the receive loop, and sessions take turns writing; the same test finished 8 of 8.

A visitor that stops reading. With one receive loop it holds the others; with per-session credit it holds only itself.

Evidence: a stuck reader no longer stalls the other sessions

Setup: The F1 test: one visitor stops reading a 1 GiB download, then 8 concurrent 1 MiB downloads to the same agent must finish within 10 s; relay, agent and target in one process over loopback TLS; same-replica and cross-replica, with the agent's data connections already open (warm) or not (cold); 3 runs per case; one Mac (Apple M5, 10 cores) at load average 14 (a kind soak was running); October 10, 2026.
Numbers: gRPC: cold 0 of 8 (same and cross-replica), warm 8 of 8 same-replica and 6 of 8 cross-replica. A gRPC stream per session fixed it at +10.1 KiB and +4 goroutines per session. link: 8 of 8 in every case, and the paused session resumes intact.
Reproduce: (a) No released command reproduces it on its own. The same property is tested in public code, though not as the F1 test: in github.com/nfltr/link, go test -race -run TestMuxV2StuckReaderIsolation .; in github.com/nfltr/relay-core, go test -race -run TestTCPTunnelStuckTargetDoesNotStallOtherSessions ./core/session/ (a service that stops reading, on one relay). (b) Our check, in the closed relay-server repository: go test -mod=vendor -race -run 'TestPeerLink|TestTCPTunnelStuckClientIsolation' ./core/replicas/; and in the closed nfltr repository, through real processes against an old and a new relay: make smoke-test-dataplane-cross-version OLD_BIN=<dir> NEW_BIN=<dir> ($0; stuck-reader isolation 8 of 8).
Limits: In process and on loopback, 8 sessions, 3 runs per case, a heavily loaded host. It shows isolation, not throughput under a stuck reader.

Per-byte overhead

A tunnel byte over gRPC is a protobuf message inside an HTTP/2 frame, with flow control twice: gRPC's and HTTP/2's. Forwarding between replicas reached 0.53 to 0.61 of same-replica throughput, with 1.8 to 2.4 times the heap. Link reached 0.95 to 0.98 of same-replica throughput for one upload session (0.90 for one download), and an idle session costs +2.3 KiB and about no goroutines.

Layers a byte crosses. A session in link is already a byte stream; over gRPC a byte stream is messages inside HTTP/2 frames.

Evidence: forwarding between replicas costs less over link

Setup: Benchmarks of a TCP tunnel within one replica and across two, 8 MiB per session, 1 and 8 sessions, upload and download; relay, agent and target in one test binary over loopback TLS; "before" is the gRPC replica-to-replica path and "after" the replica link, in the same binary; 6 interleaved runs, medians; one Mac (Apple M5, 10 cores) at load average 1.5 to 4.1 while another lane's kind soak ran; October 10 and 11, 2026 (link phases 2 and 2b).
Numbers: Cross-replica over same-replica throughput, one session: gRPC 0.53 upload and 0.61 download; link 0.95 to 0.98 upload and 0.90 download. Peak heap cross over same: gRPC 1.8 to 2.4 times. Per forwarded idle session: gRPC +10.1 KiB and +4 goroutines, link +2.3 KiB and +0.04.
Reproduce: (a) Not with released binaries: production runs one relay, so nothing you run crosses replicas. The same-replica side is public: in github.com/nfltr/relay-core, go test -run '^$' -bench BenchmarkTCPTunnelThroughput -benchmem -count 6 ./core/session/. (b) Our check, in the closed relay-server repository: go test -mod=vendor -run '^$' -bench 'BenchmarkPeerLinkThroughput|BenchmarkLinkSmallRequestLatency' -benchmem -count 6 ./core/replicas/ and LINK_PATH=cross go test -mod=vendor -run TestLinkMemoryPerSession ./core/replicas/ (also same and link).
Limits: Eight sessions reached only 0.83 to 0.86 of same-replica throughput, below our 0.9 target: the hop is a second TLS encryption and two more copies of every byte on the same 10 cores as relay, agent and target. Loopback has no latency or loss, the host was shared, and the absolute numbers (about 1,000 to 2,900 MB/s) say nothing about a real network.

Window tuning

Per-session flow control needs a window sized for the path. A 4 MiB window halved download speed at 40 ms round-trip time; 8 MiB was needed. With one flow-control layer instead of two, there is one number to tune.

Evidence: a 4 MiB window halves downloads at 40 ms

Setup: The tunnel throughput benchmark with an emulated 40 ms round trip, on gRPC streams, with the credit step at half and at a quarter of the window; the same Mac; October 2026.
Numbers: With a 4 MiB relay window, download lost 50 %; upload lost 4.5 % to HTTP/2's own window with the step at half. 8 MiB with a quarter step recovered it; 16 MiB measured the same as 8 and holds twice the memory per stuck session.
Reproduce: (a) In github.com/nfltr/relay-core (Apache-2.0): go test -run '^$' -bench BenchmarkTCPTunnelThroughputRTT ./core/session/. It runs today's data path, which is link, not the gRPC streams measured here, so expect different numbers. (b) The gRPC-era runs above were in the closed relay-server repository, before the data path moved to relay-core.
Limits: One emulated delay, no loss, one machine. A longer path needs a larger window, and link's window is fixed per configuration, not tuned automatically.

Lifecycle and drain

A rollout should let sessions in flight finish while new ones go elsewhere. Link has a goaway frame for it: the draining side says so, the peer opens no more sessions on the connection, the old ones finish, and the connection closes. In the drain test, 8 of 8 old sessions finished and 8 of 8 new ones succeeded elsewhere.

A rollout from the connection's point of view.

Evidence: a drain loses no old session and refuses no new one

Setup: The drain case of link's comparison rig (below): a dialer pool already at its connection limit, 8 sessions open, the listener drains, then 8 new sessions are opened; dialer and listener in separate containers; link v0.4.2; October 11, 2026.
Numbers: 8 of 8 old sessions finished with their last bytes and 8 of 8 new ones succeeded on a replacement connection. On v0.4.1, before the fix, old sessions finished 8, 7 and 6 of 8 and new ones 0 of 8 (the pool was full).
Reproduce: (a) In github.com/nfltr/link (Apache-2.0): make compare-docker CASES=drain LIBS=link (docker with NET_ADMIN), then make compare-summary; go test -run TestGoAwayDrainKeepsTails . is the unit test. (b) Our check for the relay, in the closed relay-server repository: go test -mod=vendor -run TestPeerLinkDrain ./core/replicas/.
Limits: A rig, not a cluster: rollouts on Kubernetes also depend on how the relay records and moves agents, measured in The relay as several replicas on Kubernetes. The public repository starts at v0.10.17, so the rig now runs a later link than the one measured here.

Scaffolding

Around gRPC sat a connection pool, an expiring cache, a circuit breaker, token metadata on a separate port, and keepalive tuning. Link brings the pool and the liveness itself, and the change deleted about 4,000 lines.

What it does not fix

TCP head-of-line blocking is shared by every TCP transport. Under 1 % packet loss a pool of 4 connections was 3.5 to 5 times faster than one. Only QUIC removes it.
Failed visits during a rollout and the time to recover from a network partition were not gRPC's doing; they belong to how the relay records and drains agents, and are separate work.

Evidence: under loss, four connections beat one

Setup: Two relay replicas in containers, 8 cross-replica echo sessions, a replica-link pool of 1 against 4 connections, mutual TLS and plaintext, no netem and 20 ms delay with 1 % loss; 3 interleaved runs each, 15 s per run, medians; October 11, 2026.
Numbers: Mutual TLS under loss: 3.5 MB/s with one connection, 12.3 MB/s with four; plaintext 2.6 against 13.3 MB/s. Without loss one connection carried what four do, within 2 %, at about 10 % less CPU per byte. The default stayed at 4.
Reproduce: (a) No public harness runs two relay replicas; link's comparison rig in github.com/nfltr/link runs link alone under delay and loss (make compare-docker LIBS=link). (b) Our check, in the closed nfltr repository: make test-replica-link-faults CASES=netem NEW_RELAY=<linux nfltr-relay> NFLTR=<linux nfltr> ($0, docker) measures throughput under 2 ms / 0.1 %, 20 ms / 1 % and 50 ms / 2 % with the default pool of 4. The one-connection side was a one-off run of the same rig with the pool limit lowered; no checked-in case switches it.
Limits: One loss rate and one delay, 3 runs, one machine; the visitors and target were Python, which bounded the totals without loss.

Where each fits

	gRPC	Stream multiplexer (link)
Unit of work	Typed request and response, or a message stream	Raw byte streams, many at once
Contract	Protobuf schema, generated clients	None; the caller defines the bytes
Tooling	Rich: interceptors, load balancers, reflection	Small: sessions, credit, liveness, drain
Long-lived tunnels	Possible, with the problems above	What it is built for: many long-lived streams on one connection
Slow reader	Can hold a shared stream	Holds only its session

One agent, two kinds of connection to the relay. Both are opened by the agent.

nfltr uses both. gRPC carries control, the mailbox, presence and the APIs: small typed calls where a schema helps. Link carries the agent data connections and the forwarding between replicas: byte streams where isolation and fast failure detection matter. It also carries the sessions of the connect sidecar, which lets a remote client reach a service on an agent with no inbound port: Every service reachable from every remote client, with no inbound ports.

Keepalive presets

Link's liveness is set by a preset name rather than by raw numbers (the agent's flag now also takes interval=<duration>,timeout=<duration> for a network no preset fits). Idle cost is one small frame each way per interval per connection, and a dead peer is found within interval plus timeout.

Preset	Interval / timeout	Dead peer found in
fast	1.5 s / 3 s	4.5 s
default	5 s / 10 s	15 s
nat	10 s / 20 s	30 s
datacenter	15 s / 30 s	45 s
mobile	15 s / 45 s	60 s

A preset beats two raw knobs because it names the network once. The dialer states it and the listener adopts the dialer's, so the two ends cannot disagree, and the interval is bounded to 1 to 15 s so a typo cannot turn pings into a flood or into silence. On the agent it is the --keepalive flag or NFLTR_KEEPALIVE; in the Helm chart, peerLink.liveness sets it for the replica link.

The trade-off is false alarms. In our loss tests the fast preset declared a dead peer wrongly in 1 of 8 runs at 0.1 CPU with loss; the default never did. A reasonable choice: default for most, fast inside a datacenter where a quick failover matters and loss is rare, nat for agents behind devices that expire idle mappings, datacenter or mobile for links with long pauses or flaky radios. Common combinations are default on the replica link with agents on default, fast, nat or mobile according to where each runs.

Evidence: the fast preset can raise a false alarm under starvation

Setup: The false-dead-peer case of link's comparison rig: 50 ms delay and 2 % loss, the containers limited to 0.25 and 0.1 CPU with GC pressure, 60 s busy (8 sessions) then 60 s idle, fast and default; 4 runs per pass, two passes on link v0.4.2; October 11, 2026.
Numbers: fast: 1 false dead peer in 8 runs (pass 1, at 0.1 CPU; that side lost all 8 sessions), 0 in pass 2. default: 0.
Reproduce: In github.com/nfltr/link (Apache-2.0): make compare-docker CASES=falsedead LIBS=link, then make compare-summary.
Limits: Eight runs is a small sample for a rare event; 0.1 CPU is harsher than most hosts, and the relay's own fault matrix saw no false alarm with either preset at 0.25 and 0.1 CPU. The public repository starts at v0.10.17, so the rig now runs a later link than the one measured here.

link and yamux

yamux is the established Go multiplexer, so we ran the same tests against both before trusting ours.

	link	yamux
Throughput, clean network, 8 sessions on one connection	3 to 15 % more over mTLS, 17 to 47 % more in plaintext	Baseline
Throughput under loss	Ahead at 2 ms, a tie at 20 to 50 ms (TCP-bound)	Tie at 20 to 50 ms
Heap per session after moving 1 MiB each way	9 KiB / 7 KiB	992 KiB / 1.66 MiB
CPU per MB	Equal over mTLS, 18 to 38 % less on the sender in plaintext	Baseline
One reader stopped for good	8 of 8 other transfers intact	8 of 8, a tie
False dead peer under loss and CPU limits	0 of 4, and 0 of 12 fairness runs	2 of 4, and 3 of 12 fairness runs lost a connection
Connection killed mid-transfer	Sessions end with an error	A cut stream reads as a clean EOF
Dead peer found, default preset	15 s busy; up to 20 s idle in the worst case	13 s
Drain with the pool full	Was worse (below), fixed	Old and new sessions all fine
Maturity	Days old, 10 fuzz targets	12 years in production, widely imported: yamux leads

Evidence: link against yamux on the replica link's job

Setup: link v0.4.1 against hashicorp/yamux v0.1.2 (its latest release), each behind the same pool, header and drain layer, with matched settings (1 MiB per-session window, 5 s / 10 s and 1.5 s / 3 s liveness, TLS 1.3 with mutual certificates or plaintext TCP, 64 KiB writes). Two processes on one Mac (Apple M5, 10 cores), and docker containers with pinned CPUs, netem for delay and loss, iptables for partitions and CPU limits. 5 repetitions (3 for fault cases), medians, the two libraries interleaved; the machine was shared, load average 0.8 to 13.7; October 11, 2026. The fault rows were re-run on link v0.4.2.
Numbers: Clean network, 8 sessions: link 3 to 15 % ahead over mutual TLS, 17 to 47 % in plaintext. Heap per session after 1 MiB each way: 9 / 7 KiB against 992 KiB / 1.66 MiB. False dead peer under loss and CPU limits: 0 of 4 against 2 of 4. A cut connection: an error in link, a clean EOF in yamux. Dead peer found: 15 s against 13 s.
Reproduce: In github.com/nfltr/link (Apache-2.0), its compare/ module: make compare (two processes on one host), make compare-docker (docker with NET_ADMIN, about an hour), then make compare-summary.
Limits: One machine, shared with other work, so CPU-bound rows carry noise. yamux ran inside a layer we wrote for this job (pool, header, drain, dead-peer events parsed from its log), which a yamux user might write differently. Only yamux was compared here; the API review in part 2 adds smux, quic-go and libp2p, without measurements. The public repository starts at v0.10.17, so the rig now runs a later link than the one measured here.

The decisive difference is the cut connection. A stream cut off by a dead connection can look to a reader like a stream that ended normally. For a file or a database dump that is silent corruption, and link reports an error. yamux is ahead on detection speed as configured and on maturity. The comparison also exposed four defects in link, among them a dead peer that took up to 20 s to be found when idle against its own 15 s bound, and old sessions lost and new ones refused when a drain met a full pool. All are fixed in link v0.4.2.

How link was made

From evidence. The Kubernetes soak and the F1 test showed the stuck reader and the dead-peer hang, and raised the question of why one IP-to-IP link needed two connection codes.
Phase 1 extracted the existing protocol byte for byte, checked against golden recorded bytes, so nothing changed on the wire.
Wire version 2 added credit, goaway, liveness presets and reset. The replica link was built on it and the gRPC path was deleted.
A fault matrix of kills, stalls and loss found a 30 s write-starvation bug.
The yamux comparison found four more.
The v2.0.0 hard cut: client generation 2, wire version 1 dropped, older clients refused with a message to run nfltr update.

It was built test-first by parallel agent lanes, each with measured targets, on 10 and 11 October 2026.

What is next

The 6-hour multi-replica soak has since passed, with the replica link built on the same contract other code can now program against: three replicas, 66 agents and rolling restarts, pod kills, Redis restarts and network partitions in rotation. Every kind of traffic held 100 % steady availability (p99 217–235 ms), about two thirds of it crossing replicas over link; mailbox delivery stayed exactly once across 21,585 messages; goroutines and Redis keys did not grow. Rolling restarts failed 2–3 visits each, down from 17 before a drain bug found by the first attempt was fixed. One of three partitions took 26 s to recover, above the 10 s target; that is the next thing being measured. How the relay behaves as several replicas, and what it took to get there, is in The relay as several replicas on Kubernetes.

Evidence: the replica link held a 6-hour multi-replica soak

Setup: A kind cluster on one Mac: 3 relay replicas (relay-server v0.15.0, link v0.6.1), 66 agents, Redis and Postgres in the cluster; a fault every 20 minutes in rotation, 3 each of rolling restart, pod kill, Redis restart and network partition; one 6-hour run (the second attempt; the first stopped on a harness fault), October 12 to 13, 2026.
Numbers: 100 % steady availability for every kind of traffic, p99 217–235 ms, 65–67 % of visits crossing replicas; 21,585 mailbox messages, 0 duplicates, 0 lost; goroutines net of agents 361 to 337, Redis keys 184 to 181. Rolling restarts: 2–3 failed visits each, recovered in 18–20 s. Partitions recovered in 26.3, 0 and 2.9 s.
Reproduce: (a) No released binary or public repository runs this soak. (b) Our check, in the closed nfltr repository: SOAK_DURATION=6h make soak-relay-kind ($0; docker and kind; each run makes and deletes its own cluster). make soak-relay-kind-short is the 25-minute version.
Limits: One run, on one machine, with a load spike from another process (1-minute load up to 39) overlapping part of it. "Steady" excludes the fault windows: each Redis restart still failed 433–520 visits and each partition 14–281. Clock-skew faults were skipped on the arm64 nodes. kind's network is not a real network between machines.

At the time, link was not a public repository, and publishing it waited on a 24-hour soak, fuzzing, six weeks in production and a security review. It became public under Apache-2.0 on 2026-10-29 (what is open) and is still at v0; part 2 describes the API it froze its way towards.

Users on an older client need to update: nfltr update --with orch --apply. Clients older than v2.0.0 are refused.

Is it for you?

Fits if you run nfltr agents and want their TCP tunnels, end-to-end encrypted shares and proxied HTTP to survive a slow visitor and to notice a dead relay in seconds: link is already under every one of them, and the --keepalive preset is the one knob. Also if you write Go and need many byte streams over one connection with per-stream credit, drain and an error on a cut stream: link is a library of its own with no dependencies outside the standard library.

Not ready for anyone who needs a mature multiplexer: link is weeks old against yamux's twelve years, still v0, and its stream reset carries no error code. The multi-replica relay has run on kind and once on a cloud cluster, but nfltr.xyz runs one relay on one VM, and your own relay is a beta of one process. The cross-replica benchmarks above live in the closed relay-server repository; outside readers can rerun link's own rig and relay-core's same-replica tunnel benchmarks. Under packet loss link is as limited by TCP as any TCP transport; the QUIC agent link is opt-in.

Try next nfltr update --with orch --apply, then pick a preset for where an agent runs, for example nfltr http 3000 --keepalive nat behind a home router (flags in the CLI reference). To run the relay yourself, see Run your own relay. link's code, its comparison rig and its soak are at github.com/nfltr/link; the API decisions behind its v1 are in part 2. Where link sits among the other public repositories is in Four nfltr repositories are now open source, and the relay's earlier data-path work, before link, is in A faster, leaner relay.