You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ghc: retry hadrian once on the stamp-file lock race (GHC #24365) - #1357
In the pkgs#1351 (zlib 1.4) world build, ghc failed in stage1 on amd64 with no compile or link error:
_build/stage1/libraries/os-string/build/stamp-os-string-2.0.7-inplace_dyn: withFile: resource busy (file is locked)
Build failed.
The same build passed on arm64. This is a known intermittent hadrian race. It appears when hadrian runs in parallel, and we have built that way since #575 (-j$(nproc)). Upstream tracks it as GHC #24365, "Intermittent hadrian exception: withFile: resource busy (file is locked)" (https://gitlab.haskell.org/ghc/ghc/-/issues/24365). That issue reports the same stamp-<pkg>-inplace_dyn files failing on Linux and Darwin CI. It is still open and no GHC release fixes it, so moving to a newer GHC would not help. GHC #24382 (https://gitlab.haskell.org/ghc/ghc/-/issues/24382) covers a Windows variant from copyFile. GHC #20886 notes that withFile can put the wrong file name on an exception, so the stamp name might be misleading, but the failure is transient either way.
Change
Both hadrian invocations now go through a small run_hadrian helper in packages/ghc/build.sh. It tees hadrian's output to hadrian-<phase>.attempt<N>.log next to _build. If the first attempt fails and that log contains resource busy (file is locked), the helper prints "hadrian (): transient stamp-file lock race; retrying once (Shake resumes)" and runs hadrian again. Every other case fails as it did before: a different error, or the lock error a second time, returns hadrian's exit code with no retry.
The install invocation goes through the same helper because hadrian install re-checks the whole build graph under the same -j before copying, so it can hit the same race. HADRIAN_FLAGS now gets passed inside the helper, which means the build and the install still use identical flags, as #575 requires.
Reproducibility
Flags, the flavour and the install prefix are unchanged, and nothing new is written to $OUTPUT_DIR. A resumed build produces the same bytes. Shake loads .shake.database and skips every rule that already completed, so those outputs stay on disk untouched. It reruns only the rules that did not finish, with the same commands and flags. The two #277 patches are applied to the source before hadrian runs, so they are unaffected.
Verification
bash -n packages/ghc/build.sh passes.
A shell harness ran the real block from build.sh (from HADRIAN_FLAGS thr
8000
ough the install line) against a stubbed _build/bin/hadrian:
Lock error once, then success: one retry, then the install runs. 3 hadrian calls, exit 0.
A different error: no retry, build aborts. 1 call, exit 1.
Lock error twice: one retry, then the build aborts. 2 calls, exit 1.
Install hits the lock error once: the install retries once and passes. DESTDIR reaches both install calls.
I did not run a local ghc build because it is long and blocked by sandbox egress (minimal#3058). The build-bot will exercise this change.
Rebuild scope
The ghc spec changes, so these rebuild:
ghc
packages that import it directly: cabal, stack, haskell-language-server, tamarin-prover, dev-language-servers
packages that depend on those: cabal pulls in stack, haskell-language-server and tamarin-prover, and haskell-language-server pulls in dev-language-servers
That is 6 packages in total, all Haskell. ghc has build_cost_multiple = 6, so this is best landed alongside a world rebuild rather than on its own.
Build and installation operations now retry once when they encounter a specific file-lock error. Other failures, including a repeated file-lock error, still return a failure status.
Parallel hadrian intermittently fails on a stamp file it is writing itself
("stamp-os-string-2.0.7-inplace_dyn: withFile: resource busy (file is
locked)", amd64, pkgs#1351 world build). Upstream GHC #24365, still open.
Run both hadrian invocations through run_hadrian, which tees the output to
a log and retries exactly once when that log contains "resource busy (file
is locked)". Any other failure, or the lock error twice, fails as before.
Shake resumes from .shake.database, so completed rules are not rerun and
flags and outputs are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No actionable comments were generated in the recent review.
Walkthrough
The build script adds a wrapper that logs Hadrian invocations and retries once when the first failure log contains resource busy (file is locked). Both build and install use the wrapper.
Changes: Hadrian retry handling
Layer / File(s)
Summary
Retry behavior and build/install integration packages/ghc/build.sh
The script logs each Hadrian invocation. It retries once only when the first failure log contains the exact lock error text; other failures and a failed retry return their exit status. Build and install use the wrapper, with install retaining DESTDIR and the /usr prefix.
Build and install retry the captured lock error once. No material merge risk is established.
Pre-merge checks: 5
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title clearly describes the Hadrian retry for the lock race.
Docstring Coverage
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage
Explanation
Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files.
Finishing Touches
Generate docstrings
Commit to this branch
Create a new PR
Generate unit tests (beta)
Commit to this branch
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Reviewing files that changed from the base of the PR and between 7a198f0 and c8ea89c.
Files selected for processing: 1
packages/ghc/build.sh
Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 1 review per hour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the pkgs#1351 (zlib 1.4) world build, ghc failed in stage1 on amd64 with no compile or link error:
The same build passed on arm64. This is a known intermittent hadrian race. It appears when hadrian runs in parallel, and we have built that way since #575 (
-j$(nproc)). Upstream tracks it as GHC #24365, "Intermittent hadrian exception: withFile: resource busy (file is locked)" (https://gitlab.haskell.org/ghc/ghc/-/issues/24365). That issue reports the samestamp-<pkg>-inplace_dynfiles failing on Linux and Darwin CI. It is still open and no GHC release fixes it, so moving to a newer GHC would not help. GHC #24382 (https://gitlab.haskell.org/ghc/ghc/-/issues/24382) covers a Windows variant from copyFile. GHC #20886 notes thatwithFilecan put the wrong file name on an exception, so the stamp name might be misleading, but the failure is transient either way.Change
Both hadrian invocations now go through a small
run_hadrianhelper in packages/ghc/build.sh. It tees hadrian's output tohadrian-<phase>.attempt<N>.lognext to_build. If the first attempt fails and that log containsresource busy (file is locked), the helper prints "hadrian (): transient stamp-file lock race; retrying once (Shake resumes)" and runs hadrian again. Every other case fails as it did before: a different error, or the lock error a second time, returns hadrian's exit code with no retry.The install invocation goes through the same helper because
hadrian installre-checks the whole build graph under the same-jbefore copying, so it can hit the same race.HADRIAN_FLAGSnow gets passed inside the helper, which means the build and the install still use identical flags, as #575 requires.Reproducibility
Flags, the flavour and the install prefix are unchanged, and nothing new is written to
$OUTPUT_DIR. A resumed build produces the same bytes. Shake loads.shake.databaseand skips every rule that already completed, so those outputs stay on disk untouched. It reruns only the rules that did not finish, with the same commands and flags. The two #277 patches are applied to the source before hadrian runs, so they are unaffected.Verification
bash -n packages/ghc/build.shpasses.HADRIAN_FLAGSthr 8000 ough the install line) against a stubbed_build/bin/hadrian:DESTDIRreaches both install calls.Rebuild scope
The ghc spec changes, so these rebuild:
That is 6 packages in total, all Haskell. ghc has
build_cost_multiple = 6, so this is best landed alongside a world rebuild rather than on its own.🤖 Generated with Claude Code
Summary by CodeRabbit