Relay metrics on a laptop, through nfltr

October 10, 2026

nfltr connects machines through a relay they dial out to, so a port that listens only on a server's loopback can still be reached from a laptop, by the right key, without opening it. The nfltr.xyz relay exports Prometheus metrics on such a port. We wanted graphs and alerts over weeks without opening that port to the internet, without a hosted monitoring service, and without copying the data anywhere but the operator's laptop. Most small servers have the same problem: a metrics endpoint you should not expose, and no wish to run a monitoring stack next to it.

What we did and found: an owner-only agent on the server, nfltr tcp-connect on the laptop, and Prometheus and Grafana in Docker next to it. The setup is shown with placeholders, so it works for any server with a Prometheus endpoint. The one real mistake we nearly made was with labels: dropping a per-agent label in Prometheus alone makes the connected-agents gauge read 1, so the sidecar sums the series before dropping the label. Grafana's first memory limit was also too tight. Some of this has changed since, and the updates are marked.

The shape

• On the server, an nfltr up agent with one TCP service that forwards to the metrics port on loopback. It connects out to the relay; nothing listens on a public address.
• On the laptop, three containers: a small sidecar that runs nfltr tcp-connect to that service and serves the metrics to Prometheus, Prometheus itself, and Grafana. Their host ports bind to 127.0.0.1 only.

Only keys of the account that owns the agent reach it. A TCP service gets no share URL, and the agent dials only the targets in its file. The scraped data lives in a Docker volume on the laptop.

On the server

A config file for nfltr up, with your metrics port in place of <port>:

agent: <server>
services:
  metrics: { tcp: <port>, forward: "127.0.0.1:<port>" }

nfltr up -c nfltr.yaml

On nfltr.xyz this is one service of the agent we use for operator access to the VM, described in nfltr on nfltr, part 2. Run it as a service with its key in a file only that service can read, not on a command line.

One detail to know: the relay resolves the agent name in a tcp-connect request inside the calling key's own namespace. If the laptop and the server use the same key, or the server's key was minted on the dashboard under the laptop key's name, the laptop reaches the agent by name. If they use two unrelated keys of the same account, it does not yet.

Update: since relay-server v0.4.0 (October 10, 2026) a connect request may name the full agent ID of any agent its account owns, so two keys of the same account work; a short name still resolves in the calling key's namespace first.

Check it by hand

nfltr tcp-connect <agent>/metrics --listen :9182
curl -s localhost:9182/metrics | grep -v '^#' | head

If that prints samples, the tunnel works. Everything below automates it.

On the laptop: three containers

A trimmed version of our compose file. The sidecar image is Debian slim with curl and socat, and nfltr installed by the public install.sh, which verifies its signature. The API key is a 0600 file mounted as a compose secret, so it is never in the image, the compose file or a command line.

name: metrics
services:
  sidecar:
    build: .
    environment:
      METRICS_AGENT: <agent>
    secrets: [{source: nfltr_env, target: nfltr.env}]   # NFLTR_API_KEY=...
    cpus: 0.25
    mem_limit: 128m
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:<version>@sha256:<digest>
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=30d
      - --storage.tsdb.retention.size=2GB
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports: ["127.0.0.1:9090:9090"]
    cpus: 0.5
    mem_limit: 512m
    restart: unless-stopped

  grafana:
    image: grafana/grafana:<version>@sha256:<digest>
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
      GF_AUTH_ANONYMOUS_ENABLED: "false"
      GF_USERS_ALLOW_SIGN_UP: "false"
    secrets: [grafana_admin_password]
    volumes: [grafana-data:/var/lib/grafana]
    ports: ["127.0.0.1:3000:3000"]
    cpus: 0.5
    mem_limit: 640m
    restart: unless-stopped

secrets:
  nfltr_env: {file: ./secrets/nfltr.env}
  grafana_admin_password: {file: ./secrets/grafana-admin-password}

volumes:
  prometheus-data:
  grafana-data:

The sidecar's entrypoint loads the key from the secret file into its environment, then runs two things and exits when either stops, so Docker restarts the pair:

set -a; . /run/secrets/nfltr.env; set +a
nfltr tcp-connect "$METRICS_AGENT/metrics" --listen :9182 &
socat TCP-LISTEN:9180,fork,reuseaddr EXEC:/usr/local/bin/serve-metrics &
wait -n

Update: since v2.1.6 (October 16, 2026) our sidecar runs nfltr up with a connect: mapping to the metrics service instead of a bare tcp-connect, with the service end-to-end encrypted and the agent's certificate pinned, so the relay carries only ciphertext; the rest of the shape is unchanged. The mapping's fields are in the nfltr up reference, and how a connect: mapping works is in Every service reachable from every remote client, with no inbound ports.

serve-metrics answers one scrape: it fetches /metrics through the tunnel, passes it through a short awk filter (next section), and answers 503 when the server is unreachable, so Prometheus marks the target down instead of storing nothing quietly.

# prometheus.yml
global:
  scrape_interval: 30s
  scrape_timeout: 25s
scrape_configs:
  - job_name: <job>
    static_configs:
      - targets: [sidecar:9180]
    metric_relabel_configs:
      - action: labeldrop
        regex: agent_id|owner|account|user|email|ip|remote_addr|peer|key_id|tenant

Sum the labels, then drop them

nfltr.xyz serves other accounts, and the relay labels some series by agent with agent_id, which holds other accounts' agent names. Those names should not be stored on our laptop. The obvious fix is a labeldrop in prometheus.yml. On its own it is wrong.

Series that differ only in agent_id become identical once the label is dropped. Prometheus then sees duplicate series in one scrape and keeps one sample, not their sum. The connected-agents gauge has one series per agent, so with a labeldrop alone it would read 1 however many agents were connected, and every per-agent counter would undercount the same way.

So the sidecar sums first. Its awk filter removes the identifying labels from each sample and adds up the samples they told apart; counters, gauges and histogram buckets, sums and counts all add. The labeldrop in prometheus.yml stays, with the same list, as a second line of defence, and a test checks the two lists are equal. The test's own fixture shows the difference (pod_id is kept: it tells relay replicas apart, as in The relay as several replicas on Kubernetes):

# in
c{agent_id="other.acct",pod_id="p"} 1
c{agent_id="mine.vm",pod_id="p"} 1
h_bucket{agent_id="a",route="/{$}",le="+Inf"} 2
h_bucket{route="/{$}",agent_id="b",le="+Inf"} 3

# out
c{pod_id="p"} 2
h_bucket{route="/{$}",le="+Inf"} 5

Our list covers agent_id, which is the only identifying label the relay exports today, plus account, user, email, address, key and tenant names that it might add later. When it adds one, it goes into both lists, and a status check fails if Prometheus stores any listed label name.

Summing has a cost. When an agent disconnects, its share of a summed counter goes with it, and rate() reads the drop as a counter reset. Where the server exports an unlabelled total, such as HTTP requests or process network bytes, the dashboard uses that instead.

Update: from relay-server v0.4.0 the relay labels no series with an agent, account or key: per-agent series are totals and connected agents is one gauge, so the sum has nothing left to do. The labeldrop stays as the guard, and the status check still fails if a listed label is ever stored.

Dashboard and alerts

Grafana starts with a provisioned Prometheus data source and one dashboard: scrape health, uptime, connected agents, goroutines and memory; HTTP request rate, errors and p95 latency by route; gRPC streams by method and code; throughput; TCP, SSH, SNI, terminal and agent-to-agent sessions; garbage collection, CPU and open files. Anonymous access, sign-up and update checks are off.

Prometheus evaluates alert rules with no notifier: the target down for five minutes, a restart, goroutines, heap or memory growing over six hours, more than 5 % HTTP 5xx, and gRPC errors above 20 % of finished streams. They show at localhost:9090/alerts. The thresholds are starting points.

The same tunnel also carries Go's profiler when the server enables it:

go tool pprof http://localhost:9182/debug/pprof/heap

A CPU profile costs the server CPU while it runs, so it is only for when you need it.

What went wrong, and the limits

• Grafana's first memory limit was too tight. It idled at 332 MiB of a 384 MiB limit, so the limit is now 640 MiB. Prometheus stays at 512 MiB and the sidecar at 128 MiB.
• The metrics share the relay's fate. The agent connects through the relay it reports on. If that relay is down, the target goes down, an alert fires, and you look at the server another way; for nfltr.xyz that is the cloud provider's SSH. For a server that is not the relay itself, this does not apply.
• No version metric. The relay exports no build-info metric, so the dashboard shows Go's version; the relay's own version is at https://nfltr.xyz/api/v1/version. (Update: newer relays export nfltr_relay_build_info, and the dashboard's build stat shows the relay version from it.)
• The label list is maintained by hand. A new identifying label on the server is stored until someone adds it to both lists; the check catches only names already on the list.
• It runs while the laptop does. A sleeping laptop leaves gaps in the graphs.

For the tunnel commands themselves, SSH to a machine behind NAT shows nfltr tcp-connect with other TCP services, such as a database.

Is it for you?

Fits if you have a server with a Prometheus endpoint (or Go's pprof) on loopback, want graphs and alerts on your own laptop with no open port and no hosted monitoring, and are fine with gaps while the laptop sleeps.

Not ready for always-on monitoring or paging: Prometheus here has alert rules but no notifier, and it runs only while the laptop does. Reaching an agent across keys needs relay-server v0.4.0 or later. When the server is the relay itself, the metrics go down with it. Our compose files and dashboard are in the nfltr repository, which is closed source; the shapes above are what you get.

Try next on the server nfltr up -c nfltr.yaml with one tcp service forwarding to your metrics port, on the laptop nfltr tcp-connect <agent>/metrics --listen :9182 and a scrape of localhost:9182; then the compose file above. SSH to a machine behind NAT covers the tunnel commands, and nfltr on nfltr, part 2 the access agent this rides on. If the server you watch is a relay you run yourself, see Run your own relay.