8000 fix: bound large repository tree memory and cleanup by tdurieux · Pull Request #918 · tdurieux/anonymous_github · GitHub
[go: up one dir, main page]
Skip to content
{"props":{"contextRegion":{"crumbs":[{"crumb_type":"user","label":"tdurieux","is_root":false,"href":"/tdurieux"},{"crumb_type":"repository","label":"anonymous_github","is_root":false,"href":"/tdurieux/anonymous_github"}],"localNavigation":[{"id":"code","icon":"code","label":"Code","href":"/tdurieux/anonymous_github","selectedLinks":["repo_source","repo_downloads","repo_commits","repo_releases","repo_tags","repo_branches","repo_packages","repo_deployments","repo_attestations"],"popoverTarget":false,"commandId":"repositories:go-to-code","reactNav":{"appTarget":"code-view","anchor":"code-view-repo-link"},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"issues","icon":"issue-opened","label":"Issues","href":"/tdurieux/anonymous_github/issues","selectedLinks":["repo_issues","repo_labels","repo_milestones"],"count":15,"popoverTarget":false,"commandId":"repositories:go-to-issues","reactNav":{"appTarget":"repo","anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"pull-requests","icon":"git-pull-request","label":"Pull requests","href":"/tdurieux/anonymous_github/pulls","selectedLinks":["repo_pulls","checks"],"count":0,"popoverTarget":false,"commandId":"repositories:go-to-pull-requests","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"discussions","icon":"comment-discussion","label":"Discussions","href":"/tdurieux/anonymous_github/discussions","selectedLinks":["repo_discussions"],"popoverTarget":false,"commandId":"repositories:go-to-discussions","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"actions","icon":"play","label":"Actions","href":"/tdurieux/anonymous_github/actions","selectedLinks":["repo_actions"],"popoverTarget":false,"commandId":"repositories:go-to-actions","reactNav":{"appTarget":"repo","anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"projects","icon":"table","label":"Projects","href":"/tdurieux/anonymous_github/projects","selectedLinks":["repo_projects","new_repo_project","repo_project"],"popoverTarget":false,"commandId":"repositories:go-to-projects","reactNav":{"appTarget":"repo","anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"security-and-quality","icon":"shield","label":"Security and quality","href":"/tdurieux/anonymous_github/security","selectedLinks":["security","overview","alerts","policy","token_scanning","code_scanning"],"count":1,"popoverTarget":false,"commandId":"repositories:go-to-security","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}},{"id":"insights","icon":"graph","label":"Insights","href":"/tdurieux/anonymous_github/pulse","selectedLinks":["repo_graphs","repo_contributors","dependency_graph","dependabot_updates","pulse","people","community"],"popoverTarget":false,"commandId":"repositories:go-to-insights","reactNav":{"appTarget":null,"anchor":null},"turboNav":{"frame":"repo-content-turbo-frame"}}],"localNavigationUpdateChannel":null,"selectedLink":"repo_pulls"},"owner":null,"headerLogo":{"href":"/","aria-label":"Homepage "},"notifications":{"indicatorMode":"disabled","websocketChannel":null,"fetchIndicatorSrc":"/notifications/indicator","fetchIndicatorEnabled":false},"issues":{"href":"/issues"},"pulls":{"href":"/pulls"},"contributedRepos":{"href":"/repos"},"copilot":{"show":false,"showAgentsButton":false,"showRelaunchAnnouncement":false,"copilotChatUrl":null,"copilotApiUrl":"https://api.githubcopilot.com"},"search":{"show":true,"showCommandPalette":false,"isSearchPage":false,"isJumpToSearch":false,"searchContext":{"scope":"repo:tdurieux/anonymous_github","current_repo_name":"anonymous_github","current_repo_nwo":"tdurieux/anonymous_github","user_id":"tdurieux"}},"commandPalette":null,"enterpriseBar":{"show":false},"globalTransactionalMessage":[],"payloadsUrl":"/_global-navigation/payloads.json?can_toggle_site_admin_and_employee_status=0\u0026is_admin_mode_on=0\u0026is_ui_opted_out=0\u0026show_ui_opt_out=0\u0026v=8","contextRegionUrl":"/_global-navigation/context-region.json"}}

fix: bound large repository tree memory and cleanup - #918

Merged
tdurieux merged 6 commits into
mainfrom
fix/large-repository-heap-cleanup
Nov 3, 2026
Merged

fix: bound large repository tree memory and cleanup#918
tdurieux merged 6 commits into
mainfrom
fix/large-repository-heap-cleanup

Conversation

@tdurieux
@tdurieux tdurieux commented Nov 2, 2026 •
edited
Loading
Copy link
Copy Markdown
Owner

Large repositories can exhaust the V8 heap while downloading their tree or listing files. Refresh also waits for metadata deletion across millions of rows before enqueueing a replacement.

Consume GitHub tree pages as plain metadata and persist at most 500 entries per batch under build ownership guards. Preparation no longer reloads the completed tree. Bound listings and folder counts at 10,000 entries and pending traversal folders at 100,000, returning HTTP 413 when exceeded.

Preparation resets atomically retire old generations and persisted path-index keys. Maintenance deletes metadata in batches of at most 500 IDs and resumes interrupted cleanup. Expiration and removal check lifecycle ownership between batches; activation limits foreground cleanup. Source replacement keeps proposed settings separate until cleanup completes, rejects stale removal claims before changing status, and accepts options-only edits during preparation.

Validation:

  • The previous tree reader exhausts a 256 MiB V8 heap on the synthetic 2,000,200-entry fixture. The new reader completes the same fixture under that limit.
  • 236 focused local tests passed, including real MongoDB lifecycle/source/settings races and the actual download processor. Independent Codex validation passed 316 focused tests.
  • TypeScript, ESLint and diff checks passed. All four CI checks passed on bb366b3: Lint, Node 22, Node 24 and consent transactions.
  • GitHub Codex and fresh independent Codex round 5 are clean on that exact SHA. The existing review thread is resolved.

Targets main after #919 merged. Its message-only progress writes preserve publication timestamps. Integration commit 97e5b50 includes current main, with build/lint and 263 focused tests passing; fresh CI is running. Run npm run migrate:performance when deploying for the retired path-index maintenance index. Physical deletion still scales with row count; preparation no longer waits for that metadata deletion. No production deployment performed.

@tdurieux
tdurieux commented Nov 2, 2026
Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector
chatgpt-codex-connector Bot commented Nov 2, 2026 •
edited
Loading
Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T22:18:49.160398Z bb366b3 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@tdurieux tdurieux mentioned this pull request Nov 2, 2026
Merged
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Nov 2, 2026
View reviewed changes
@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd05781eb5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/core/Repository.ts Resolved
@tdurieux
tdurieux force-pushed the fix/large-repository-heap-cleanup branch from fd05781 to 314d961 Compare November 2, 2026 21:57
@tdurieux
tdurieux changed the base branch from main to fix/download-progress-fences November 2, 2026 21:57
@tdurieux
tdurieux commented Nov 2, 2026
Copy link
Copy Markdown
Owner Author

@codex review

Please review current head 314d961. This PR now stacks on #919, whose message-only progress updates fix the previous statusDate finding while preserving lifecycle fences. The latest heap/cleanup changes also capture and retire the persisted path-index key with compare-and-swap retries during reset and activation, including concurrent-search regression tests.

@chatgpt-codex-connector
chatgpt-codex-connector Bot commented Nov 2, 2026
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 314d961960

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@tdurieux
tdurieux commented Nov 2, 2026
Copy link
Copy Markdown
Owner Author

@codex review

Please review current head e96d235, stacked on #919. The latest fix fences tree retirement against the captured lifecycle, source/access/settings/metadata revisions and applicable refresh lease before mutating any state. Stale workers now cancel without retiring archived metadata. All 194 focused tests pass, including archive-before-reset and archive-between-key-read/CAS races, source/access changes, expired/re 8000 placed leases and valid guarded removal. Earlier persisted-key and progress-timestamp findings are addressed.

@chatgpt-codex-connector
chatgpt-codex-connector Bot commented Nov 2, 2026
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: e96d235cc7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@tdurieux
tdurieux commented Nov 2, 2026
Copy link
Copy Markdown
Owner Author

@codex review

Please review latest head abfe153, stacked on #919. Latest fix keeps proposed owner source/access/anonymization changes separate until removal under the persisted snapshot completes. Two real-Mongo save-route regressions reproduced repository_changed on the previous head and now pass for ordinary source changes and reconnects. All 221 focused tests plus TypeScript/ESLint pass; final full suite and CI running. Prior persisted path-index CAS, lifecycle retirement fence and queue progress timestamp findings are addressed. Please review the full current delta and source-change/removal interactions.

@chatgpt-codex-connector
chatgpt-codex-connector Bot commented Nov 2, 2026
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: abfe153cdc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@tdurieux
tdurieux commented Nov 2, 2026
Copy link
Copy Markdown
Owner Author

@codex review

Please review latest head bb366b3, stacked on #919. Two independent-review races are fixed: guarded removal now checks the full cached-source/lifecycle snapshot plus settingsSavedAt before claiming REMOVING, and preparation accepts options-only saves without cancelling its job. All source/access/generation/metadata/lifecycle/lease reset fences remain; expiry still checks settings changes. New real-Mongo regressions reproduce both failures on abfe153 and pass now, including the actual save route and download processor. All236focusedtests plus TypeScript/ESLint pass. Please review the whole current delta and prior persisted-key, lifecycle, source replacement, progress and settings-save fixes.

@chatgpt-codex-connector
chatgpt-codex-connector Bot commented Nov 2, 2026
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: bb366b370b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@tdurieux
tdurieux commented Nov 2, 2026
Copy link
Copy Markdown
Owner Author

Validation complete for bb366b3.

All four CI checks passed: Lint, Node 22, Node 24 and consent transactions. GitHub Codex and fresh independent Codex round 5 found no actionable issues on this exact SHA. The existing inline review thread is resolved. Independent validation passed 316 focused tests; local validation passed 236 focused tests. The earlier abfe153 full local suite passed 1,686 tests with 25 skipped.

Stack dependency: #919 at 19b1d5b. Deployment requires the additive performance-index migration documented in this PR. No merge or deployment performed.

Base automatically changed from fix/download-progress-fences to main November 3, 2026 08:22
@tdurieux ChatGPT Codex Connector
tdurieux commented Nov 3, 2026
Copy link
Copy Markdown
Owner Author

Retargeted to main after #919 merged and integrated main in 97e5b50, including #917 and #915. Resolved the documentation conflict by retaining both the large-repository and shared-cache sections. TypeScript/build and lint pass; 263 focused tests pass, including MongoDB lifecycle races, progress fences, large-tree handling, and shared-cache integration. Fresh CI is running. The documented additive performance-index migration is still required at deployment.

@tdurieux
tdurieux merged commit 99a2bff into main Nov 3, 2026
4 checks passed
@tdurieux
tdurieux deleted the fix/large-repository-heap-cleanup branch November 3, 2026 08:32
@tdurieux tdurieux mentioned this pull request Nov 3, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tdurieux
0