gRPC over the connect sidecar: drop client keepalive, keep deadlines

October 17, 2026

A gRPC client that crosses networks is usually tuned for that network: client keepalive pings so a NAT or a load balancer does not drop an idle stream, and so a dead peer is noticed before TCP gives up minutes later. Those settings are easy to get wrong. Ping too often and the server answers with a GOAWAY that ends every call on the connection; rely on pings alone and a stuck server looks healthy. nfltr connects machines through an outbound-only relay, and its connect sidecar gives the application a local socket for a remote service. Once a channel runs through it, the application's connection ends at that socket, and the network between machines is a chain of link connections that watch themselves. This post is about which settings still matter then.

We measured each claim with grpc-go against a private relay, real sidecars and injected faults. With no client keepalive, a dead path ended an idle stream in 14.9 s. A 10 s client keepalive drew the server's too_many_pings in 31 s. A stuck handler outlived 45 s of keepalive, and a 2 s deadline ended it. One hop still needs care, a forward: to another host: a silent host took 153 s to surface with Go's default TCP keepalive (now 25.6 s), and only a client keepalive finds a frozen one (80 s). So: drop client keepalive, keep deadlines, with that one exception.

Who watches which hop

A call through a connect: mapping crosses four hops. The application talks to a local socket. The sidecar and the agent each hold a link connection to the relay. The agent dials the server.

The sidecar never parses HTTP/2. gRPC's own PING frames, deadlines and RST_STREAMs are bytes it carries, so everything gRPC does end to end still works, including the server's enforcement of how often a client may ping.

Deadlines and cancellation pass through

On one channel we made 20 unary calls, opened a stream and made 5 more calls. The sidecar's /status counted one session for all of it: one gRPC channel is one HTTP/2 connection, and that is one session to one service.

• A call with a 3,000 ms deadline reached a handler that saw 2,999 ms left. grpc-timeout is a header, and nothing on the way touches it.
• A 1 s deadline on a handler that never returns: the client got DEADLINE_EXCEEDED at 1,002 ms and the handler's context ended at 1,002 ms.
• A cancel after 500 ms: CANCELED at the client, the handler's context ended at 500 ms.

The handler's context ended with "canceled" in both cases: the client's RST_STREAM arrived before the server's own timer from grpc-timeout fired. That is what grpc-go does without a sidecar too.

A dead path ends the call in 15 seconds, with no keepalive

We opened an idle server stream with no client keepalive configured, then dropped every packet between two machines with iptables: no FIN, no RST, the worst kind of failure.

Each link connection sends a small ping once nothing has arrived for 5 s and declares its peer dead 15 s after its last frame. When a hop dies, the sessions on it are reset both ways, so the sidecar resets the application's local connection and gRPC returns UNAVAILABLE. The server's handler sees its context end. Neither side hangs until a TCP retransmission timer gives up minutes later.

Relay restarts and drains

A relay that leaves (a deploy, a restart) drains: new sessions go elsewhere, and open ones keep their bytes until the drain deadline. We restarted the only relay under an open stream, with a 10 s drain window.

• The stream ended UNAVAILABLE 10.1 s after SIGTERM, and the sidecar's /status gave the reason: drain_deadline.
• A retried idempotent call succeeded 11.6 s after SIGTERM, on a new session, once the relay was back (4 attempts, 500 ms apart). In an earlier run it took 15.7 s.
• Right after a relay-to-agent path came back, we also saw the first opens of a mapping fail UNAVAILABLE for about two seconds before one succeeded. The same retry covers it.

With several relay replicas, new sessions open on another replica at once and only the sessions still open at the deadline are cut.

Client keepalive is optional, and can hurt

Keepalive pings are gRPC's own frames, so the server still counts them, sidecar or not. grpc-go's default server allows one ping every 5 minutes on a connection with streams and none without, and answers a client that pings faster with a GOAWAY (ENHANCE_YOUR_CALM, too_many_pings), which ends every call on the connection.

• Client keepalive 10 s with pings allowed without streams, on an idle stream: GOAWAY too_many_pings after 31 s. The client's error said so: received prior goaway: code: ENHANCE_YOUR_CALM, debug data: "too_many_pings".
• Client keepalive 60 s, timeout 20 s, on a healthy idle stream: GOAWAY too_many_pings after 181 s.

The sidecar hops need no help: link already found the dead paths above in 15 s. A keepalive that was set so a NAT or a load balancer would not drop an idle connection has nothing to protect any more, because the application's connection now ends at a local socket.

Keepalive does not find a stuck server. A deadline does

A server whose handler is stuck, waiting on a lock or a dead dependency, still has a healthy HTTP/2 transport, and that transport answers PINGs. We gave the client keepalive 10 s with a 5 s timeout, against a server that permits it, and called a handler that never returns:

• After 45 s the call was still open and the channel READY; 115 bytes of PING ACKs had come back in that time.
• The same call with a 2 s deadline ended DEADLINE_EXCEEDED at 2.0 s, and the handler's context ended at the same moment.

Keepalive answers "is the connection alive", and link answers that already. Only a deadline answers "is my call going to finish".

The exception: forward: to another host

An agent can publish a server that runs on another host, or behind a load balancer: grpc: { tcp: 50051, forward: "ledger.internal:50051" }. That last hop is a plain TCP connection from the agent, and no link watches it. We read what the agent's dialer set: net.Dialer{Timeout: 10s} with no KeepAlive, so Go's defaults applied: TCP keepalive on, first probe after 15 s idle, then every 15 s, dead after 9 unanswered probes (ss showed the socket's keepalive timer). That took 153 s to notice a silent host, so the agent now sets its own: first probe after 10 s idle, then every 5 s, dead after 3 (relay SDK client/v0.17.4). We made that host vanish under an idle stream in two ways: iptables DROP on the agent (a host gone silently), and docker pause (a frozen host whose kernel still answers TCP):

So for an idle long-lived stream to a forwarded server, the agent's TCP keepalive finds a vanished host in about 25 s, and only a client keepalive finds a frozen one. A 60 s keepalive needs the server to allow it, though: against grpc-go's default enforcement it drew too_many_pings after 181 s. Keep the client's keepalive time at or above what the server enforces (5 minutes for a default grpc-go server), or lower the server's minimum where you own the server. Calls with deadlines need none of this; it matters only for streams that stay idle for minutes.

The sidecar points at what to change

When the application speaks gRPC over plaintext HTTP/2 to its mapping, as most do on a loopback or user-only socket, the sidecar reads the frame headers as they pass (never a body) and its /status says what is left to change. After the runs above, the 10 s keepalive's mapping showed:

"inspect": "h2c",
"hints": [{"kind": "client_keepalive",
           "message": "relax or drop client keepalive: it must be ≥ the server's enforcement minimum (grpc-go: 5 min), ...",
           "evidence": {"idle_pings_under_15s": 2, "too_many_pings_goaway": 1}, ...},
          {"kind": "no_deadline", "evidence": {"calls": 2, "calls_without_deadline": 1}, ...}]

The other kinds are connection_per_call (a new channel per call: reuse one) and app_not_reading (the remote's data waited over 5 s on the application's flow-control window). /metrics counts them per mapping and kind, and gRPC calls by status code; a path is never a label. The application's own TLS is opaque to the sidecar, which then gives no hints; inspect: false turns reading off. The kinds and their evidence are listed above; our internal connect-sidecar guide ("Hints") keeps the same list.

Recommended settings

In Go, that is a channel without grpc.WithKeepaliveParams and a deadline on every call:

conn, _ := grpc.NewClient("unix:///run/user/1000/nfltr/ledger.sock",
    grpc.WithTransportCredentials(insecure.NewCredentials()))
// no keepalive: link watches the hops through the relay

ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // every call
defer cancel()
resp, err := client.Get(ctx, req)
if status.Code(err) == codes.Unavailable {
    // retry if Get is idempotent: a new session opens
}

Migrating checklist

1. Point the channel at the mapping's local endpoint. The stubs, interceptors and TLS options stay.
2. Remove client keepalive that was there for the network: NAT or load balancer idle timeouts, a VPN. Keep it only for streams idle for minutes to a forward: server on another host, at or above the server's enforced minimum.
3. Put a deadline on every call, streams included, sized for the work.
4. Keep the retry on UNAVAILABLE for idempotent calls, with backoff.
5. Leave the server's keepalive enforcement and MaxConnectionAge as they are.
6. Keep connect timeouts short; the sidecar fails a call at once when the remote is offline.
7. Watch the sidecar's /status and /metrics: last_error.reason says drain_deadline when a relay drain cut a session, and the mapping's hints name a keepalive, a missing deadline, a channel per call or an unread stream that is left.

We keep the same guidance in the closed nfltr repository's guides, next to the code (the hops, the settings, this checklist and the forward: exception); this post is its public copy. Every flag is in the nfltr up and nfltr dial reference.

Across several link hops

The path above is a chain: the sidecar's link to the relay, then the relay's link to the agent, joined at the relay. Longer chains (several relays, your own hops built on link) behave the same way, hop by hop.

Liveness is per hop.

• Each link connection pings only while nothing arrives, and declares its peer dead within its own interval plus timeout. The cost is per connection, not per session: an idle link carried 156 bytes in 20 s with one session on it and 208 bytes with 500.
• Each hop has its own setting. The sidecar's connection uses link's default (dead within 15 s), an agent sets its own (keepalive: in its nfltr up file), and a listener with no setting of its own adopts the dialer's.
• When a middle link dies, the hops on both sides of it detect it within that link's bound and send ABORT both ways, with CodeConnLost and the index of the hop it came from. The ends read an error, never a clean end of stream. In process, with the default preset on every link of a three-link chain and the middle link black-holed, the origin read it after 14.0 s and the far end after 15.0 s: the failed link's bound plus one trip along the chain.
• A drain does the same at its deadline with CodeDrainDeadline; that is the drain_deadline reason above.

What per-hop liveness cannot see: a process that is stuck while its connection still answers pings (a deadline finds it, as above), the plain TCP forward: hop, and a reader that stopped reading, which per-session flow control holds to its own session instead.

Deadlines across hops. link carries no deadline value from hop to hop. Each side's deadlines are its own, and giving up travels as an abort or a close, which every hop passes on.

• The open is bounded by the context the origin passes: Answered(ctx), Pool.Open(ctx, …) or Node.Open(ctx, …) covers the open across every hop. A hop can add a per-hop limit, ForwardConfig.AnswerTimeout, which refuses with CodeTimeout and its hop index. nfltr's relay sets none, so the origin's context (the sidecar's --open-timeout) governs.
• SetReadDeadline and SetWriteDeadline are local, as on a net.Conn. A forwarding hop applies no read deadline and no idle limit; a write deadline set on a spliced session aborts both ways with CodeTimeout.
• For gRPC, grpc-timeout carries the end-to-end budget in band; the handler two links away saw 2,999 ms of 3,000 ms.

link does not add a deadline of its own because the application's protocol already carries one where it is needed (gRPC, HTTP), a wall-clock deadline would need every hop to agree on the time or subtract its own delay, and the case that matters, a side giving up, already reaches every hop as an abort.

For Go code on link directly

Programs that use link themselves get the same behaviour from four calls:

• Pick a liveness preset per connection on the dialing side: LivenessFast (dead within 4.5 s), LivenessDefault (15 s), LivenessNAT (30 s), LivenessDatacenter (45 s), LivenessMobile (60 s), or link.ParseLiveness("nat,timeout=40s") from a flag. A dead peer surfaces as link.ErrDeadPeer, or across hops as a *link.Abort with CodeConnLost.
• Bound the open with a context: Open(ctx) on a Pool or Node, or Answered(ctx) after Mux.Open.
• Set deadlines on the session with SetReadDeadline and SetWriteDeadline; a passed read deadline is os.ErrDeadlineExceeded, exactly as on a net.Conn.
• Propagate giving up with Abort(code, msg) or Close, and put the budget in the header when the far end needs it.

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
s, err := m.Open([]byte("5s")) // the budget, if the far end needs it
if err != nil {
    return err
}
if err := s.Answered(ctx); err != nil { // the open, across every hop
    return err
}
stop := context.AfterFunc(ctx, func() { _ = s.Abort(link.CodeApp+1, "caller gave up") })
defer stop()
_ = s.SetReadDeadline(time.Now().Add(200 * time.Millisecond)) // this side only

This is adapted from link's Example_deadlines, which go test runs: the far end, two links away, reads the origin's abort as *link.Abort with the application's code and Hops 2. Its sibling Example_deadMiddleHop checks the dead-middle-link behaviour above on every test run.

Is it for you?

Fits if you run gRPC through the connect sidecar (or are about to, as in part 2) and carry keepalive settings that were tuned for NATs, load balancers or a VPN. The settings table and checklist above are the whole change for most clients. Go programs that use link directly get the same model from the four calls in the last section.

Not ready for conclusions about other gRPC implementations: only grpc-go was measured, on one host, mostly in one run per claim, and the faults were clean drops and pauses rather than lossy real networks. The forward: hop is plain TCP and is still yours to watch. link has been open source since October 29, 2026, so its Example_deadlines and Example_deadMiddleHop run in a public clone; the harness behind the other numbers, the nfltr CLI and the relay server are closed source.

Try next Remove grpc.WithKeepaliveParams from a client that dials a mapping, add a deadline to every call, and give the client's sidecar a status endpoint (status: "127.0.0.1:7801" in its file, then nfltr up -c app.yaml) so curl -s http://127.0.0.1:7801/status shows the mapping's hints and last errors. Every flag is in the nfltr up and nfltr dial reference; part 1 covers where the sidecar fits, link: why relay replicas stopped talking gRPC how the hops watch themselves, and Designing a stream multiplexer API the deadline and liveness calls a Go program uses on link directly. To try link itself: go run github.com/nfltr/link/examples/quickstart@latest.