Repository navigation
Relayed peer never recovers after its WireGuard connection drops ("already relayed + holepunch failing" has no recovery path) #186
Description
Describe the Bug
If a site peer is already relayed when its WireGuard connection drops, olm never recovers it. The peer stays connected: false until olm restarts. In our case it stayed down for over 20 hours.
checkHolepunchEndpoints in peers/monitor/monitor.go only acts in two cases:
if !result.Success && !isRelayed && failureCount >= maxAttempts {
// trigger relay
} else if result.Success && isRelayed {
// switch back to direct
}There's no branch for !result.Success && isRelayed. When the relayed tunnel dies (for example because the connection to the server and the relay was briefly lost) and the direct endpoint still doesn't answer the holepunch test, olm does nothing. It doesn't re-request a relay, doesn't re-handshake, and doesn't escalate. When the server later re-syncs the site, the handshake handler returns early with Peer with site ID N already added, so that doesn't rebuild the tunnel either.
The same logic is in 1.10.1 and on main (eee8873) as far as I can tell.
In the same incident, two other sites on the same olm that were not relayed recovered normally through local fallback → holepunch → relay → direct.
olm logs for the stuck site (site 3) during the incident. Times are UTC+1. The network blip was at about 16:02:
WARN: 2026/11/04 16:02:48 WireGuard connection to site 3 is DISCONNECTED
INFO: 2026/11/04 16:05:18 Updated monitor endpoint for site 3 to 100.90.128.5:55216
INFO: 2026/11/04 16:05:18 Successfully updated peer for site 3
WARN: 2026/11/04 16:05:18 Peer with site ID 3 already added
After that there are no further log lines for site 3: no holepunch status changes, no relay or unrelay. By contrast, site 1 (which was using a local endpoint) logged falling back to public/relay logic → triggering relay → CONNECTED.
/status from the olm API socket about 20 hours later:
"3": {"siteId":3,"name":"staging-bastion","connected":false,"rtt":0,
"lastSeen":"2026-11-05T12:48:14+01:00","endpoint":"<site-public-ip>:55215",
"isRelay":true,"isLocal":false,"holepunchConnected":false}On the site side, newt reconnected to Pangolin within about 2 minutes (Tunnel connection to server established successfully!) and stayed healthy. Clients of other sites kept working, and traffic from the site to other resources was fine. Only this olm → site tunnel stayed down.
After systemctl restart of olm, site 3 connected straight away (connected: true, isLocal: true, isRelay: false).
Environment
- OS Type & Version: NixOS 26.05 (olm, Pangolin server and site hosts)
- Pangolin Version: 1.22.2
- Edition (Community or Enterprise): Community
- Gerbil Version: 1.5.1
- Traefik Version: 3.7.10
- Newt Version: 1.16.0
- Client Version: olm 1.8.2 (CLI, Linux,
--override-dns=false). The code path is unchanged in 1.10.1.
To Reproduce
I don't have a deterministic reproduction. This is what happened, plus a suggested way to force it:
- Run olm connected to a site where the direct holepunch path fails, so olm relays it (
isRelay: true,holepunchConnected: false). - Briefly break connectivity between olm, the Pangolin server/Gerbil and the site, long enough for the WireGuard connection check to report
DISCONNECTEDand for newt to lose its websocket. Ours was about 2 minutes. Blocking UDP to Gerbil's relay port on the olm host for a couple of minutes should have the same effect. - Restore connectivity. newt reconnects and the server re-syncs the site to olm (
Peer with site ID N already added). - The relayed peer stays
connected: falseindefinitely, and olm logs nothing more for it. Non-relayed peers on the same olm recover.
We saw the same blip on 2026-10-30, and that time the site recovered after about 8 minutes. I don't have olm status from that time, so I can't confirm it was relayed then.
AI Disclosure
I used an AI assistant to correlate the logs, read the olm source around checkHolepunchEndpoints, and draft this issue. I reviewed the analysis and the logs myself.
Expected Behavior
A relayed peer whose WireGuard connection stays disconnected should be recovered automatically. For example, after a timeout olm could re-request a relay, tear down and re-add the peer, or retry direct and local endpoints. Restarting the client shouldn't be necessary.
Activity
AstralDestiny commentedon Nov 5, 2026
Are you using pangolin cli or olm directly?