Repository navigation
Conversation
hsorbo
marked this pull request as draft
November 6, 2026 17:36
TcpConnection.read() blocked its caller until the lwIP thread had copied the data out and called tcp_recved(), costing two cross-thread wakeups per read. This made both the TLS tunnel stream and relayed streams latency-bound. Copy out under the state lock instead, which on_recv() already uses, and let lwIP know about the consumed data asynchronously. Limit each read to what a single tcp_recved() call can account for.
Waiting for handle_incoming_datagrams() and an idle iteration before reading more meant that decryption and lwIP processing never overlapped. The netstack processes requests in order, so there's no need to wait.
handle_incoming_datagrams() is now synchronous and thread-safe, queuing to the lwIP thread (or writing to the TUN device) without waiting for the result, so drivers can feed it straight from their I/O threads. A netstack can also deliver outgoing datagrams to a handler on the producing thread instead of bouncing them through the main context.
Every USB transfer completion used to bounce through the main context, wait for lwIP, and only then get resubmitted; outgoing datagrams went the other way through two idle callbacks. Each hop added a wakeup on a busy main loop to the round-trip the device sees, so even with the netstack read path fixed, downlink through our NCM driver stayed at a third of what the kernel's cdc_ncm driver got. Receive transfers are now resubmitted and parsed on the libusb thread and handed straight to the netstack, and outgoing datagrams are packed and submitted from the thread producing them. Transfers keep the driver alive and count as device operations, and UsbDevice.close() cancels whatever is still in flight before closing the handle, so teardown no longer depends on the device happening to send something.
hsorbo
force-pushed
the
lwip-read-path
branch
from
November 6, 2026 18:07
8abcefa to
9700971
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keep the main loop out of the tunnel data path
Tunnel throughput on Linux is bound by main-loop round-trips, not CPU: on the host below, downlink through a CoreDevice tunnel sat at 70–90 Mbit/s with the kernel NCM driver or usbmuxd, and at ~30 Mbit/s with our own NCM driver, while the host was mostly idle. This PR removes the round-trips. It comes in two parts, two commits each.
Part 1:
lwip:andtunnel:(small, low risk)TcpConnection.read()no longer blocks on the lwIP thread. It copies out under the state lock thaton_recv()already uses, and tells lwIP about consumed data asynchronously.Net −21 lines, no interface changes. This part works standalone: if the full PR is too big or too risky, I can split it out into its own PR. On its own it roughly triples downlink in all three transport modes.
Part 2:
netstack:andncm:Lets the userspace NCM driver bypass the main loop entirely:
VirtualNetworkStack.handle_incoming_datagrams()becomes synchronous and thread-safe (queues to the lwIP thread, or writes to the TUN device, without waiting), and a netstack can deliver outgoing datagrams to a handler on the producing thread instead of through the signal on the main context.UsbDevice.close()now cancels the device's in-flight transfers before closing the handle. Before, the stop path only completed because completed receive transfers failed to re-allocate an operation while flushing, i.e. it depended on the device happening to send something.Benchmarks
Host: Linux (Ubuntu 26.04.1, kernel 7.0.0-42-generic) on an Intel Pentium N4200 (4 cores, 1.1 GHz), USB 2.0. Device: iPad, iOS 17.7.11.
All three stages were run with #8 applied, so that idund isn't spinning; the spin keeps a core out of deep C-states and inflates numbers.
Mbit/s, down / up / bidir (down + up):
Baseline (
main+ #8)cdc_ncmAfter part 1
After part 2 (whole PR)
Uplink is unchanged throughout:
write()still round-trips to the lwIP thread, which could be a follow-up.Testing
openChannel, verified word by word on the host, in usbmuxd and userspace-NCM modes.idundwithCAP_NET_ADMIN) still works.While testing teardown I noticed that the USB transport, tunnel and
NcmPeerare never freed (pre-existing, refcount 4 after close, so~NcmPeer()never closes the driver). This PR makes teardown independent of that, but the leak itself is left for a separate change.