Repository navigation
Requests to a newt site hang ~85 s after the newt reconnects from a hard crash #127
Description
We run two newt sites behind the same resources, so a connector host can die without taking the resources down. Related to fosrl/pangolin#3855, but a separate problem: that one is about the dead site staying online, this one is about what happens after the dead newt comes back.
When a newt's host dies hard (no clean shutdown) and the newt later reconnects, requests Traefik routes to that site hang for about 85 seconds and then fail, even though the new tunnel works. It clears on its own. A clean newt restart doesn't do this.
Setup: self-hosted Pangolin EE 1.24.0, gerbil 1.5.2, Traefik v3.7 sharing gerbil's network namespace (the standard compose), newt 1.18.1. Two Pangolin servers; both showed the stall, the capture below is from one.
What happens
We hard-stopped the newt's container at 21:05:10 UTC and started it again at 21:11:34. The newt reconnected at 21:11:41 and gerbil swapped the peer:
21:11:41 Clearing connections for removed peer with WG IP: <site-ip>
21:11:41 Cleared 0 connections for WG IP: <site-ip>
21:11:41 Peer <old-key> removed successfully
21:11:41 Clearing connections for added peer with WG IP: <site-ip>
21:11:41 Cleared 0 connections for WG IP: <site-ip>
21:11:41 Peer <new-key> added successfully
From then until 21:13:07, 40% and 20% of our probe's requests failed on the two servers. In Traefik's access log every failed request routed to that site ends at the same instant, 21:13:07.
A capture on wg0 inside gerbil's namespace shows why. Traefik still had a pooled HTTP/2 connection to <site-ip>:<target-port>, opened to the old newt before the stop. Nothing ever closed it, so Traefik kept putting new requests on it while its TCP retransmits backed off. The next retransmit went out at 21:13:07.186, the new newt answered with a RST at 21:13:07.213, and Traefik opened a fresh connection at 21:13:07.217. Everything worked from there. Meanwhile the new tunnel was fine: other targets on the same site opened new connections and exchanged data normally during the stall.
What we ruled out
The path. Full-size WireGuard packets flowed both ways during the stall, with no relay involved and no ICMP from the newt's side. A clean newt restart, which closes its connections, recovers within about 20 seconds. Traefik's HTTP/2 ping health check would catch this, but it's off by default and can only be set per service, and Pangolin only emits a per-service transport when a resource has a TLS server name.
Reproduction
A resource with a target on a newt site, steady traffic to it, hard-stop the newt's host (we used pct stop on its LXC host), wait a few minutes, start it again. Requests on that site hang until Traefik's next retransmit on the old connection, which after a few minutes of backoff is a minute or more away.
From reading relay/relay.go on main, OnPeerAdded/OnPeerRemoved close gerbil's own UDP relay connections for the WG IP, but not TCP sockets other processes in the namespace hold to it. Would it make sense for gerbil to also kill TCP sockets to a WG IP when its peer is replaced or removed (like ss -K dst <ip>), or is that better handled somewhere else?
(The capture, the measurements and the code reading were done with an AI coding agent I run. The log lines are verbatim apart from addresses and keys. Happy to test a patch.)
Activity