8000 Add relay servers for rollback matches by inada-s · Pull Request #428 · inada-s/gdxsv · GitHub
[go: up one dir, main page]

Skip to content

Add relay servers for rollback matches - #428

Merged
inada-s merged 6 commits into
masterfrom
gcp-relay
Sep 30, 2026
Merged

inada-s merged 6 commits into
masterfrom
gcp-relay

Conversation

@inada-s
@inada-s inada-s commented Sep 29, 2026 •
Copy link
Copy Markdown
Owner

Summary

Add relay servers that rollback players can route their GGPO packets through when their direct route is poor.

Some long-distance pairs (e.g. Japan ↔ Hong Kong) get a detour through Singapore on certain ISP pairs. From real match logs, the direct RTT between them swings between ~75 ms and ~175 ms depending on the ISP pair and the day, which sometimes pushes the input delay to 4–5 frames. A relay on GCP Premium Tier reaches both sides through Google's backbone. A relay VM in asia-east2 reached the Hong Kong ISPs in 2–3 ms and Japanese players in ~65 ms, and a real UDP relay test matched the sum of the legs (the relay added ~0 ms). The existing peer relay can't help when every remote player is on the far side.

  • gdxsv relay (relay.go): forwards GGPO packets. Forwarding takes ~22 ns per packet and allocates nothing, so a busy relay does not make the GC run (sessions and bindings are keyed by netip.AddrPort; a test checks for 0 allocations). It forwards them using GGPO's existing peer-relay header (type 99 + relay_to_endpoint). Players join a session with a small UDP ping carrying session id, peer id and a per-match token, so only bound members of the same session can relay to each other. It registers with the lobby like mcs (new lbsExtRelayStatus / lbsExtRelayControl). The status carries GDXSV_RELAY_SECRET, and the lobby accepts only relays with the same secret (compared in constant time, and not kept in the stored status). Otherwise anyone who connects to the lobby could register as a relay and sit in the middle of matches. It exits when the lobby says so, or after losing the lobby for 15 minutes. -relay_test_sessi 8000 on=<id>:<hex token> runs it without a lobby for local tests.
  • The lobby's own relay: the lobby also runs a relay on GDXSV_RELAY_ADDR (UDP 9879 by default, empty to disable). It is free and always there, so a match can use a relay before any VM is up.
  • Relay VMs (lbs_relay.go), enabled by GDXSV_RELAY_REGIONS (comma separated; VMs only ever start in these regions):
    • While at least 4 players whose clients support relays are online, from at least two region groups, start a relay VM through mcsfunc in every listed region. A player's region group is the existing gcpRegionGroup of their best GCP region, the same one the region-friendly team shuffle uses. So Japan + Hong Kong starts relays; Japan only, Japan + Korea, or Hong Kong + Taiwan does not.
    • Stop a relay after it has been unneeded for 30 minutes and no match is using it.
    • The check only walks the online players, so it costs next to nothing in the event loop. The region list lets another region (e.g. Osaka) be added if one is not enough.
  • Per match: when every player reports relay_server=1, the match gets every connected relay (up to 4), best first. Relay VMs come first, ordered by their worst estimated relayed RTT between participants, then the lobby's own relay. Each relay gets the match's session and a fresh token (P2PMatching.relays). Clients pick a relay per peer and settle disagreements on the earlier one.
  • IPv6: relays listen on IPv4 and IPv6 and pass their IPv6 address as RelayServer.ip6, so IPv6 and IPv4 players can reach each other through one. Clients ping both families. A relay keeps one address per family for each player and accepts both. It sends packets for a player to the address that player last sent a game packet from, so the family the player actually plays over keeps its NAT mapping. The addresses come from the machine's own interface (not an external lookup), skipping temporary and deprecated ones: in the relay VM startup script, and in launch-lbs.sh for the lobby's relay.
  • mcsfunc role=relay: starts an e2-micro VM running gdxsv relay in the custom-mode network gdxsv-relay. It passes its own GDXSV_RELAY_SECRET in the startup script, and refuses to start a relay unless that secret is set and plain ([A-Za-z0-9_-]{16,}). The VM gets external IPv4 and IPv6 and shuts itself down when the relay exits. The http option is left out for these VMs: the client library would replace their access configs and add a firewall rule to the default network.

Client side: inada-s/flycast#411

GCP setup (done)

  • Network gdxsv-relay (custom mode), subnet gdxsv-relay-asia-east2 (10.200.0.0/24, IPv4/IPv6 with external IPv6).
  • Firewall gdxsv-relay-udp4 (0.0.0.0/0) and gdxsv-relay-udp6 (::/0): udp:9879 to tag gdxsv-relay.
  • Checked with a temporary VM in that subnet: the external IPv6 GCP assigns is on the NIC, and the startup script's detection returns it.
  • The default network is untouched (auto mode, IPv4 only).

Deploy notes

  • mcsfunc is already deployed from this branch on nodejs22 (functions-framework 3.5.1; 1.9 does not start on new Node). The URL, service account and limits are unchanged. A test copy listed the mcs VMs and created a relay VM in gdxsv-relay with IPv4 and IPv6, which stopped itself when the relay exited. Until this is released, a relay VM stops right away, because the released binary has no relay command.
  • The relay VM runs the latest gdxsv release, so release this before enabling relay VMs.
  • Set the same GDXSV_RELAY_SECRET on the lobby and on mcsfunc (done on both). It is not in the repository.
  • The lobby: launch-lbs.sh now sets the relay addresses. UDP 9879 is open on the VPS for IPv4 and IPv6. Enable relay VMs with GDXSV_RELAY_REGIONS=asia-east2.

Test

  • go test -race ./... and golangci-lint: pass.
  • Unit tests cover:
    • forwarding, session binding and expiry;
    • 0 allocations per forwarded packet or ping;
    • IPv4 → IPv6 forwarding through a real dual-stack socket;
    • a player bound on both families, answered where it plays from;
    • start conditions (player count, region groups, older clients) and starting every listed region;
    • relay registration with a right, wrong or missing secret, or none on the lobby;
    • relay ordering and the cap;
    • the lobby's relay;
    • IPv6 address handling.
  • An integration test runs a relay against a real TCP lobby: registration, session with IPv6, kept while in use, shut down when idle.
  • flycast tools/rbk_test cases D4–D6 run 4 local clients through local relays: one relay, two relays settling on one, and a dual-stack relay where players with an unreachable relay IPv4 address pick IPv6 on their own. All pass.
  • Not yet tested end to end with a real lobby handing relays to clients.

Rollout checklist

Before releasing:

  • GCP: network gdxsv-relay, subnet gdxsv-relay-asia-east2 (IPv4/IPv6), firewall gdxsv-relay-udp4 / -udp6 (udp:9879).
  • mcsfunc deployed on nodejs22 with GDXSV_RELAY_SECRET.
  • Lobby VPS: UDP 9879 open for IPv4 and IPv6. launch-lbs.sh sets GDXSV_RELAY_ADDR, GDXSV_RELAY_PUBLIC_ADDR, GDXSV_RELAY_PUBLIC_ADDR6, GDXSV_RELAY_REGIONS=asia-east2 and GDXSV_RELAY_SECRET.
  • Merge, release, restart the lobby (v1.4.1, restarted 2026-10-01 01:07 UTC).

After the restart, in Cloud Logging (logs/gdxsv-lbs):

  • local relay listening shows public_addr=153.121.44.150:9879 and the VPS's IPv6 in public_addr6. There is no local relay not started, and no ignoring relay IPv6 address.
  • (VM start checked by hand, see below; the automatic trigger waits for 1.9.3 players) While at least 4 relay-capable players from two region groups are online (e.g. Japan + Hong Kong): mcsfunc alloc for role relay appears, and VM gdxsv-relay-asia-east2-latest is RUNNING with IPv4 and IPv6.
  • relay registered with region=asia-east2, and no relay status with a wrong secret.
  • Matches with relays: the players' P2PMatchingReport before_log has RELAY0 rtt=[...] lines, and Relay:2 when a peer went through a relay (see Route rollback packets through relay servers when faster flycast#411).
  • The lobby's own relay logs relay peer bound (4 per match) and relay session removed ... forwarded=N.
  • About 30 minutes after the condition is gone: relay no longer needed, shutting it down, and the VM is TERMINATED.
  • After a week, check the relay VM's hours and egress in billing.

Checked on 2026-10-01 by starting a relay VM through the production mcsfunc (/alloc?region=asia-east2&role=relay) with no players around. It came up with IPv4 and IPv6 and registered at 10:11:45 UTC with the secret. At 10:41:53 the lobby shut it down, and the VM was TERMINATED at 10:42:29.

The relay VM's own log (/var/log/gdxsv-relay.log) is not shipped to Cloud Logging. Its sessions and forwarded counts are only visible on the VM, and the gdxsv-relay network allows no SSH.

To roll back:

  • Unset GDXSV_RELAY_REGIONS so no relay VM starts.
  • Set GDXSV_RELAY_ADDR="" to turn off the lobby's own relay.
  • Older clients are never offered relays, so they are unaffected either way.

🤖 Generated with Claude Code

Some long-distance pairs (e.g. Japan <-> Hong Kong) get a poor direct
route on certain ISPs, going through Singapore and back. Measured from
real players, the direct RTT between them swings between ~75 ms and
~175 ms depending on the ISP pair and the day, while a relay on GCP
Premium Tier reaches both sides through Google's backbone and stays
around 70 ms. Flycast already relays through peers, but that cannot
avoid the slow international leg when all remote players are on the
other side.

`gdxsv relay` forwards GGPO packets between the players of a match:
- It reuses GGPO's existing peer-relay header (type 99 with
  relay_to_endpoint), restores it like a relaying peer does, and sends
  the packet to the destination peer of the sender's session.
- Players join a session with a small UDP ping carrying the session id,
  peer id and a per-match token; only bound members of the same session
  can relay to each other.
- It registers with the lobby like mcs, reports its sessions every 3 s,
  and exits when the lobby tells it to or when the lobby has been
  unreachable for 15 minutes. The VM's startup script then shuts the VM
  down.
- -relay_test_session=<id>:<hex token> runs it without a lobby, for
  local tests.

The lobby manages relays when GDXSV_RELAY_REGIONS (comma separated) is
set:
- It starts a relay VM through mcsfunc (role=relay, e2-micro) when some
  online pair is far apart (>= 50 ms through every GCP region, from the
  latencies clients report at login), an allowed region is within 10 ms
  of their best meeting point, and the relayed estimate is <= 120 ms.
- It stops a relay after its region has been unneeded for 15 minutes
  and no match is using it.
- For each match whose players all report relay_server=1, it offers one
  relay (the one with the lowest worst-pair estimate) with a fresh
  token in P2PMatching.relays. A single relay keeps both sides of a
  pair on the same server.

The relay listens on UDP 9879, inside the existing gdxsv firewall rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lobby now runs a relay itself on GDXSV_RELAY_ADDR (UDP 9879 by
default, empty to disable). It is free and always there, so a match can
use a relay before a VM is up, and pairs no VM is started for still get
one.

A match is offered every connected relay (up to 4, what clients
accept), best first: relay VMs by their worst estimated relayed RTT
between participants, then the lobby's own. Each is given the match's
session. Clients pick a relay per peer and, when two peers disagree,
settle on the earlier one, so the order is the same for every player.

Relays take IPv6 too, so a player on IPv6 and one on IPv4 can reach
each other through one:
- `gdxsv relay` and the lobby's relay listen on both families when the
  address has no host.
- A relay reports its IPv6 address (GDXSV_RELAY_PUBLIC_ADDR6), and the
  lobby passes it as RelayServer.ip6.
- The addresses come from the machine's own interface, not an external
  lookup, skipping temporary and deprecated ones.
- mcsfunc creates relay VMs in the custom-mode network `gdxsv-relay`,
  whose subnet `gdxsv-relay-<region>` is dual-stack, with external IPv4
  and IPv6, tagged `gdxsv-relay`. The `http` option is left out for
  them: the client library would replace their access configs and add
  a firewall rule to the default network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@inada-s inada-s changed the title Add an on-demand relay server for rollback matches Add relay servers for rollback matches Sep 30, 2026
inada-s and others added 4 commits September 30, 2026 10:32
Deciding which regions need a relay VM compared every pair of online
players in the lobby's event loop, parsing 22 latencies per pair. With
400 players that stalled the lobby for ~58 ms every 10 s.

- The event loop now only copies each eligible player's latencies into
  a fixed array (0.14 ms for 400 players). Pairs are checked in a
  goroutine (1.1 ms for 400), and the result comes back through the
  event loop.
- Only players whose client supports relays count, and only pairs on
  the same game disk, since only they can play each other.
- Stop as soon as every allowed region is known to be needed.
- A relay address must be a numeric IP. Resolving a name could stall
  the event loop while a match is being made.
- Start thresholds are loosened so fewer pairs that need a relay go
  without one: far apart from 40 ms (was 50), relayed estimate up to
  150 ms (was 120).
- A relay VM is stopped after 30 minutes unneeded (was 15).

The relay keys sessions and bindings by netip.AddrPort and reads and
writes with ReadFromUDPAddrPort / WriteToUDPAddrPort. Pongs reuse the
ping's buffer. Forwarding a packet takes ~22 ns and allocates nothing
(checked by a test), so a busy relay does not make the GC run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The per-pair latency heuristic that decided when to start relay VMs was
hard to follow. Replace it with a simple rule: while at least 4 players
whose clients support relays are online, from at least two region
groups (the existing gcpRegionGroup of each player's best region, as
used by the region-friendly team shuffle), start a relay VM in every
region of GDXSV_RELAY_REGIONS. The check only walks the online players,
so it needs no goroutine. GDXSV_RELAY_REGIONS stays a list, so another
region can be added when one is not enough.

Clients now ping a relay over both IPv4 and IPv6 and use the faster
family that answers. A relay therefore keeps up to one address per
family for each player and accepts packets from either. Packets for a
player go to the address it last sent a game packet from, falling back
to its last ping before it plays. The family it plays over keeps its
NAT mapping open. Forwarding still allocates nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The nodejs12 runtime mcsfunc was deployed on can no longer be deployed
to, so it moves to nodejs22, the newest GA runtime for gen1 functions.
functions-framework 1.9 does not start there (missing
./middleware/cloudevent_to_background_event), so it moves to 3.5.1.

Deployed with the same URL, service account and limits. A test copy
listed the mcs VMs and created a relay VM in gdxsv-relay with IPv4 and
IPv6, which stopped itself when the relay exited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Anyone can connect to the lobby, and the lobby hands registered relays
to players, so a stranger could register as a relay and sit in the
middle of matches. A relay now sends GDXSV_RELAY_SECRET in its status,
and the lobby accepts only relays that send the same secret. The
comparison is constant time, and the secret is not kept in the stored
status, so it never reaches the logs. A lobby without a secret accepts
no relay VM. Its own relay needs none.

mcsfunc passes its GDXSV_RELAY_SECRET to relay VMs in their startup
script. It refuses to start a relay VM unless the secret is set and
plain ([A-Za-z0-9_-]{16,}), so nothing else can end up in the script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@inada-s
inada-s merged commit 8d973d9 into master Sep 30, 2026
2 checks passed
@inada-s
inada-s deleted the gcp-relay branch September 30, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0