Repository navigation
fix(xbrl): stop retrying a document that ran out of memory - #4813
Closed
stoyanov-x wants to merge 1 commit into
Closed
stoyanov-x wants to merge 1 commit into
stoyanov-x wants to merge 1 commit into
Conversation
Closes daniel3303#4812. The per-document catch-all counts an attempt and lets the document be retried until MaxXbrlFactsAttempts. That is right for transient faults, but an OutOfMemoryException on a large filing is deterministic: every retry hits the same wall, spends the retry budget, and leaves repeated OOM rows in Errors - which looks like a data gap rather than a signal. Handle OOM explicitly and mark the document exhausted, mirroring what the fiscal-calendar branch already does for evidence that cannot change on retry. The heap is left alone: after an OOM its state is not something to reason about, so this only stops the loop. A streaming rewrite of InlineXbrlParser/StandaloneXbrlParser is the real fix (peak memory independent of filing size) - that is a larger change and belongs in its own PR.
Owner
|
I'm leaving this one until it's out of draft. Two points for when it is:
A.L.V.I.S. |
Owner
|
Closing this, for the reasons in my earlier comment. An The existing attempt budget is the safer behaviour, and #4814 removes the DOM cost on the standalone path. #4812 stays open for the inline path. A.L.V.I.S. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4812.
XbrlFactsExtractionWorkercounts an attempt on any per-document fault and lets the document be retried untilMaxXbrlFactsAttempts. That is correct for transient faults. AnOutOfMemoryExceptionon a large filing is not one: every retry hits the same wall, spends the whole retry budget, and leaves repeated OOM rows inErrors— which read like a data gap rather than a signal.This adds an explicit
catch (OutOfMemoryException)that marks the document exhausted, mirroring what the fiscal-calendar branch already does for evidence that cannot change on retry.This parks the document — deliberately, and reversibly
XbrlFactsAttempts = MaxXbrlFactsAttemptsmeans the document is never re-attempted automatically, so it will not silently retry forever or silently succeed later. To put parked documents back into rotation once the underlying cause is fixed:Why this is
Part of, notClosesMeasured on a 1.25 GiB worker cgroup: the OOM still reproduced 30 minutes after this shipped (alongside the streaming changes), with
memory.peaksitting at the cgroup ceiling. This stops the retry loop; it does not stop the OOM. The remaining levers are the worker memory limit and the inline path, which still builds two DOMs per filing.Re-arm semantics unchanged: a parked document stays parked until someone resets the counter, and the recovery path is documented above so it cannot be forgotten.