8000 ghc: deterministic link order + package.cache (reproducible Haskell) by bryan-minimal · Pull Request #277 · gominimal/pkgs · GitHub
[go: up one dir, main page]
Skip to content

ghc: deterministic link order + package.cache (reproducible Haskell) #277

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bryan-minimal merged 2 commits into from
Jun 22, 2026
Merged

ghc: deterministic link order + package.cache (reproducible Haskell) #277

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
95e8f08
ghc: deterministic link order + package.cache (reproducible Haskell)
bryan-minimal Jun 19, 2026
9555deb
ghc: guard that 'sort' is imported in ghc-pkg Main.hs (review hardening)
bryan-minimal Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions packages/ghc/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,34 @@ cd "ghc-${MINIMAL_ARG_VERSION}"
sed -i 's/extern void\* malloc();/extern void\* malloc(long unsigned int);/' utils/hp2ps/Utilities.c
sed -i 's/extern void \*realloc();/extern void \*realloc(void \*, long unsigned int);/' utils/hp2ps/Utilities.c

# Reproducibility: GHC iterates package/UnitId collections in hash-set order,
# making the linker arg / DT_NEEDED / .dynstr order in every binary AND the
# ghc-pkg package.cache non-deterministic. Code (.text/.rodata) is already
# byte-identical; this is pure ORDERING. Sort the two collections.
#
# (1) Link order — backport of upstream GHC #26838 / MR !15453 (merged for
# 10.0.1; NOT in 9.10.x; Debian ships this exact patch for 9.10.3). Restores
# the sorted-by-UnitId order GHC <= 9.6 had.
ghc_state=compiler/GHC/Unit/State.hs
grep -q 'import Data.List ( intersperse, partition, sortBy, isSuffixOf, sortOn )' "$ghc_state" \
&& grep -q 'let preload1 = nonDetKeysUniqMap (filterUniqMap (isJust . uv_explicit) vis_map)' "$ghc_state" \
|| { echo "ERROR: GHC link-order patch targets not found in $ghc_state — GHC source changed; revisit the #26838 backport." >&2; exit 1; }
sed -i 's/import Data.List ( intersperse, partition, sortBy, isSuffixOf, sortOn )/import Data.List ( intersperse, partition, sortBy, isSuffixOf, sortOn, sort )/' "$ghc_state"
sed -i 's/let preload1 = nonDetKeysUniqMap (filterUniqMap (isJust . uv_explicit) vis_map)/let preload1 = sort $ nonDetKeysUniqMap (filterUniqMap (isJust . uv_explicit) vis_map)/' "$ghc_state"

# (2) ghc-pkg package.cache — sort the .conf list before it is read + serialized
# so the post-build `ghc-pkg recache` emits a byte-identical cache regardless
# of filesystem readdir order. (No upstream fix exists; `sort` already
# imported in Main.hs.)
ghc_pkg_main=utils/ghc-pkg/Main.hs
grep -q 'confs = map (path </>) $ filter (".conf" `isSuffixOf`) fs' "$ghc_pkg_main" \
|| { echo "ERROR: ghc-pkg package.cache patch target not found in $ghc_pkg_main — GHC source changed." >&2; exit 1; }
Comment on lines +31 to +33
@edge-delta edge-delta Bot Jun 19, 2026
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guard gap: Unlike patch (1) which explicitly adds sort to the Data.List import via sed, patch (2) relies on the assumption that sort is already in scope in Main.hs. The grep guard here only verifies the target line exists — it doesn't verify sort is importable.

If sort happens NOT to be in scope (e.g., a future GHC version reorganizes imports in ghc-pkg/Main.hs), the build would fail with a Haskell compile error deep into the multi-hour Hadrian build rather than failing early at this guard.

Consider adding a guard like:

grep -q 'import Data.List' "$ghc_pkg_main" \
  || { echo "ERROR: Data.List (sort) not imported in $ghc_pkg_main — add sort import." >&2; exit 1; }

Or, more robustly, mirror the patch (1) pattern: explicitly sed in a sort import if it's not already present, so the patch is self-contained regardless of upstream import state.

# The `sort $` below needs `sort` in scope; 9.10.3's Main.hs imports it (line 78),
# but guard it so a future GHC import reorg fails here, not deep in the build.
grep -qE 'import Data.List \(.*\bsort\b' "$ghc_pkg_main" \
|| { echo "ERROR: 'sort' not imported in $ghc_pkg_main — patch (2) requires it (add a sort import)." >&2; exit 1; }
sed -i 's#confs = map (path </>) $ filter (".conf" `isSuffixOf`) fs#confs = map (path </>) $ sort $ filter (".conf" `isSuffixOf`) fs#' "$ghc_pkg_main"

Comment on lines +12 to +39
@coderabbitai coderabbitai Bot Jun 19, 2026
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Add the required build-twice reproducibility verification step.

This script improves ordering determinism, but it still does not perform the required “build twice and compare two $OUTPUT_DIR trees byte-for-byte” verification step in build.sh. Please add that explicit check before considering this reproducibility work complete.

As per coding guidelines, “Verify reproducibility by building the package twice and comparing the two $OUTPUT_DIR trees for byte-for-byte identity.”

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ghc/build.sh` around lines 12 - 39, The build.sh script applies
reproducibility patches to GHC source code but does not include verification
that these patches actually achieve byte-for-byte reproducibility. After the sed
commands that apply patches (2) to both $ghc_state and $ghc_pkg_main files, add
a verification step that builds the package twice to separate output directories
and compares them byte-for-byte using diff or similar tools to ensure the
resulting binaries and artifacts are identical. If the comparison fails, the
script should report an error indicating the reproducibility goal was not met.

Source: Coding guidelines

# Create stubs for tools ./configure / bootstrap.py checks for but aren't strictly needed
STUB_DIR="$PWD/../stub-bin"
mkdir -p "$STUB_DIR"
Expand Down
0