Repository navigation
EMFILE caused by file descriptor limit can lock up warp #1110
Description
it looks like handling EMFILE by warp:
wai/warp/Network/Wai/Handler/Warp/Run.hs
Line 390 in 3f441f3
does more harm than good.
It is supposed to mitigate file descriptor exhaustion by introducing latency instead of erroring out. But the logic is based on a heuristic only taking into account local state (ie. only information available to warp) which is not enough very often.
Consider the scenario:
- The application starts
warp - The application is close to a file descriptor limit.
warpaccepts one connectionwarpreceivesEMFILEtrying to accept the second and subsequent connections- all connection requests wait for a single connection to finish. Once it happens, one of the waiting connection requests succeeds but all other keep waiting
- Because
waitForDecreasedreads the initial counter only once, all connections are waiting for connection counter to be decreased to 0 (see: ) which might never happen.
Effectively, warp has locked itself and even increasing file descriptor limit (or releasing some file descriptors by other parts of the application) won't help.
It looks like it would be much better to simply return error upon EMFILE instead of trying to be "smart" and handle it.
Activity
[-]EMFILE caused by global file descriptor can lock up warp[/-][+]EMFILE caused by file descriptor limit can lock up warp[/+]Vlix commentedon Aug 28, 2026
Another solution might be a(n async) check that starts running which tries to grab an FD from
/dev/null(on Unix) or somewhere else (on Windows) every few milliseconds, and flips a flag once it works.Or just retrying, instead of waiting? Maybe with exponential backoff to a 1 sec cap.
mkleczek commentedon Aug 28, 2026
Yeah, I guess that's the best solution.
kazu-yamamoto commentedon Aug 31, 2026
I don't understand this part.
Since
handleFdExhaustionwaits,acceptLoopstops at this point.It doesn't seem to me that "subsequent connections" would end up "waiting for a single connection to finish."
mkleczek commentedon Aug 31, 2026
You are right - that part is indeed missing that the first
EMFILEstops the whole accept loop.Nevertheless, the issue is still there in case of long-living connections (such as HTTP 1.1 with keep-alives) -
warpheuristics prevents accepting new connections when the number of available file descriptors is increased externally.Vlix commentedon Sep 2, 2026
So shall we just remove the
check (n < n0)part and just let it loop indefinitely oneMFILE? (like it does if we don't have any active connections)... if | isErrno eMFILE -> do - handleFdExhaustion e + fdExhaustion <- readIORef fdRef + when (fdExhaustion /= FdExhausted) $ + settingsOnException set Nothing $ E.toException e + setFdExhaustion fdRef acceptNewConnectionI don't see anything immediately wrong with this. It's one thread doing a
foreverthat keeps throwing aneMFILEuntil it maybe doesn't. And it reports the exhaustion once, until it has gotten a new file descriptor.kazu-yamamoto commentedon Sep 2, 2026
It might be a good idea to slow down the accept loop if the rate of
eMFILEerrors exceeds a certain threshold. We can use the following for rate tracking:https://hackage-content.haskell.org/package/network-control-0.1.7/docs/Network-Control.html
Vlix commentedon Sep 4, 2026
How would we use the
Ratefrom that module? Would we usex <- getRateand ifx == 1we try again?(IIUC the
Rateis initialized at 1 and gets incremented during the following second, and then is reset to 1 again, right?)kazu-yamamoto commentedon Sep 5, 2026
Right.
getRatevalue is less than our threshold, recurse.threadDelaywith a duration of less than one second.Rateseems too short, let's modify the library to allow specifying the duration.Vlix commentedon Sep 24, 2026
I think I have an idea on how to implement the rate limiting.
I'll be away this weekend, and I'm starting a new job next week, but I'll try to make a PoC somewhere next week.