Repository navigation
fix(gateway): don't hold statusMu while sending - #646
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SendholdsstatusMuwhile it callssendInternal, which takesconnMu.CloseWithCodeandopentake them the other way around,connMufirst and thenstatusMu. If the connection drops while something is sending,listencallsCloseWithCodeand the two goroutines end up waiting on each other forever. After that everySend,Status()and the reconnect block too, so the bot stays offline until it's restarted.This happened to my bot last night. The type 20 channel issue (fixed by #644/#645) made it reconnect ~300 times in 40 minutes, one of those closes landed while
SetPresencewas sending, and it stayed stuck for 18 hours. The goroutine profile had these two (line numbers from master):Sendnow reads the status throughStatus()and releases the lock before sending. If the status changes in between,sendInternalstill checksg.connunderconnMuand returnsErrShardNotConnected.To confirm it, I ran
SendandCloseWithCodeconcurrently 2000 times: master hung in 6 of 8 runs, with this change it never did. I didn't add that as a test since it depends on timing. The test checks the lock order directly instead: the rate limiter runs insidesendInternalwithconnMuheld, and from therestatusMuhas to be free.