8000 cadical 3.0.1, reproducible (pinned build.hpp host); bv_decide end-to-end test by bryan-minimal · Pull Request #757 · gominimal/pkgs · GitHub
[go: up one dir, main page]

Skip to content
7 changes: 4 additions & 3 deletions packages/aeneas-latest/build.ncl
Original file line number Diff line number Diff line change
Expand Up @@ -113,9 +113,10 @@ let aeneas_sha = "ab34bd28458a08c150b7cbf1873e59af74588138" in
# bundles it; our from-source lean does not).
toolchain,
leangz,
# `bv_decide` (exercised by Bvify's own `#guard_msgs` examples during the
# library build) shells out to `cadical`; lean releases bundle it, ours
# doesn't.
# `bv_decide` shells out to `cadical`; lean releases bundle it, ours
# doesn't. The library build runs 60 solver call sites (Bvify's
# `bv_decide` examples, and BvTac's `bv_tac` examples, Tests and simp
# lemmas), so this build is the real-goal test of a cadical bump.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit · The rewritten comment keeps "lean releases bundle it, ours doesn't" as the reason for this dep. That reason is out of date, because packages/lean/build.ncl:46 lists cadical in runtime_deps, and AGENTS.md says the runtime deps of a build_deps entry reach the build. So this build gets cadical through lean. Remove the dep, or say that it is an explicit marker for the build's use of the solver.

cadical,
curl,
git,
Expand Down
21 changes: 18 additions & 3 deletions packages/cadical/build.ncl
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,23 @@ let make = import "../make/build.ncl" in
let gcc = import "../gcc/build.ncl" in
let glibc = import "../glibc/build.ncl" in

# The release Lean 4.33 bundles as `bin/cadical` (lean4 CMake INSTALL_CADICAL).
let version = "2.1.3" in
# The latest upstream release, deliberately NOT the one Lean bundles (lean4
# 4.35's CMakeLists.txt pins rel-2.1.2): we run current versions and fix what
# breaks. A newer solver cannot make Lean accept a false goal, because
# `bv_decide` checks the solver's LRAT certificate in Lean's own verified
# checker. A solver change can only make a proof fail: a rejected
# certificate, a timeout, or a changed message. Two places would show that:
# - lean's `bv_decide` standalone test is a smoke test. It checks that
# cadical is found and that lean accepts the certificates it writes for
# four goals. Three goals are small and one is Aeneas's ML-KEM
# conditional-subtraction goal, restated on plain `BitVec 32`. It also
# checks that a false goal is refuted with a counterexample.
# - aeneas-latest's library build covers real goals. `lake build Aeneas`
# runs 60 solver call sites: 9 `bv_decide` (Bvify) and 51 `bv_tac`, which
# ends in `bv_decide` (BvTac's examples, its Tests and 30 simp lemmas).
# The goals are 8- to 128-bit scalar goals. Any failure fails that build,
# and a cadical change re-keys it, so the build-bot rebuilds it.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit · The counts here (60 call sites, 9 and 51, 30 simp lemmas) describe one aeneas_sha. They go stale on the next aeneas bump, and no bump updates this file. The same count is in packages/aeneas-latest/build.ncl:117. Line 9 has the same shape with rel-2.1.2. The comment that this PR replaces went stale in this way ("The release Lean 4.33 bundles"). Say "dozens of solver call sites" here, and keep any exact count in one place beside aeneas_sha.

let version = "3.0.1" in
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium · This comment says that lean's bv_decide standalone test catches a proof that a solver change breaks. That test proves three small goals. The real consumer is aeneas-latest: its library build runs Bvify's #guard_msgs examples through cadical (packages/aeneas-latest/build.ncl:116). The PR says the bump re-keys aeneas-latest, but its verification tables list no aeneas-latest build on 3.0.1. If an example fails on 3.0.1, for example from a timeout or changed output, the aeneas-latest build fails after merge. ci_scope.py includes aeneas-latest in the scoped check, but its only test is imports_aeneas, and I could not confirm that the check rebuilds it. I did not read the Bvify examples. Build aeneas-latest on this head before merge and record the result. Also narrow the comment to what the test shows. This is one instance: lean and aeneas-latest are the only users of this package for bv_decide.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the comment claimed too much. Fixed in 33bc21f.

What aeneas-latest runs. I read the source tarball it pins (aeneas-ab34bd2, sha256 matches build.ncl) and walked the import closure of lake build Aeneas (266 modules). The build runs 60 solver call sites:

  • 9 bv_decide: 8 examples in Tactic/Conv/Bvify/Bvify.lean and 1 in Bvify/Lemmas.lean.
  • 51 bv_tac, which ends in bvDecide: 12 examples in Solver/BvTac/BvTac.lean, 9 in BvTac/Tests.lean and 30 named simp lemmas in BvTac/Lemmas.lean. The lemmas use widths up to 128 bits.

Most of these are plain examples and lemmas, not #guard_msgs. Only one of Bvify's three #guard_msgs blocks ends in bv_decide, and it checks extract_goal's output. Some goals may be closed by bv_decide's preprocessing before cadical runs, so 60 is an upper bound on solver runs. A failing call fails the library build.

aeneas-latest was already built on 3.0.1. The build-bot's minimal build check on the previous head 29b110e (check run 127023925222) lists aeneas-latest under "packages this run built", with amd64 and arm64 both passing. That is a full library build against cadical 3.0.1 on both arches. The new head gets the same.

Changes

  • cadical/build.ncl: the comment now separates the two checks. lean's bv_decide test is a smoke test: cadical is found, lean accepts its certificates and a false goal is refuted. aeneas-latest's build covers real goals and gives the count.
  • lean/build.ncl: the test gains csub, Aeneas's ML-KEM conditional-subtraction goal from Bvify.lean, restated on plain BitVec 32 so it needs no Aeneas import. With a fake cadical it fails, so it reaches the solver. With a weakened bound it gets a counterexample. It takes under a second. I measured that locally with lean 4.34.1 and its bundled cadical 2.1.2. 3.0.1 is untested until the build-bot runs the test on this head.
  • aeneas-latest/build.ncl: the comment on its cadical dep is corrected. It said Bvify's #guard_msgs; it now gives the call sites. Comment only.

The .counterexample mode from your summary is not addressed here.

{
name = "cadical",
# CaDiCaL, the SAT solver behind Lean's `bv_decide`. Official Lean releases
Expand All @@ -17,7 +32,7 @@ let version = "2.1.3" in
{ file = "build.sh" } | Local,
{
url = "https://github.com/arminbiere/cadical/archive/refs/tags/rel-%{version}.tar.gz",
sha256 = "abfe890aa4ccda7b8449c7ad41acb113cfb8e7e8fbf5e49369075f9b00d70465",
sha256 = "0a8ea563b5a25f5aa064634814edab45cc0e45111ea0f5d412a565f806fd7e11",
extract = true,
strip_prefix = "cadical-rel-%{version}",
} | Source,
Expand Down
35 changes: 34 additions & 1 deletion packages/cadical/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,41 @@
set -ex
export CXX=g++
export CXXFLAGS="-O2 -ffile-prefix-map=$(pwd)=/builddir -gno-record-gcc-switches"
# 3.x also compiles one C file (src/kitten.c) with CC/CFLAGS (CC defaults to
# gcc for CXX=g++), so it gets the same reproducibility flags.
export CFLAGS="$CXXFLAGS"
export LDFLAGS="-Wl,--build-id=none"

# Reproducibility: scripts/make-build-header.sh writes build.hpp's DATE macro,
# which lands in .rodata. Since 3.0.0 it takes the date from SOURCE_DATE_EPOCH
# (set by the sandbox, AGENTS.md), but it still appends `uname -srmn`: the
# host's hostname and kernel release. Its other inputs are already
# deterministic here: the git id is empty (no .git in the tarball), COMPILER
# is `g++ --version`, FLAGS are the CXXFLAGS above.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low · This comment calls FLAGS deterministic, but FLAGS is CXXFLAGS, and line 4 puts the absolute build directory in CXXFLAGS through -ffile-prefix-map=$(pwd)=/builddir. The prefix map does not rewrite a string literal, so that path lands in .rodata next to DATE. The output changes if the sandbox working directory ever changes, and the fail-closed check covers only DATE. packages/kissat/build.sh:4-5 lists the build directory as host data for this same class of header. Runs A and B in the PR match, so the path is stable today. Correct the comment to say that FLAGS depends on a stable build path, or extend the check to the FLAGS line. I did not read the upstream script, so this rests on the PR statement that FLAGS is our CXXFLAGS. The $(pwd) flag is common in this repo, but an embedded flag string is specific to this header shape.

# Rather than edit the upstream script, give the make step a pinned `uname`
# on PATH (kernel name and machine only, no hostname, no kernel release) and
# TZ=UTC so the epoch renders the same everywhere.
: "${SOURCE_DATE_EPOCH:?SOURCE_DATE_EPOCH unset}"
real_uname="$(command -v uname)"
shims="$(mktemp -d)"
cat > "$shims/uname" <<SHIM
#!/bin/sh
echo "\$("$real_uname" -s) \$("$real_uname" -m)"
SHIM
chmod 755 "$shims/uname"

./configure
make -j"$(nproc)" cadical
PATH="$shims:$PATH" TZ=UTC make -j"$(nproc)" cadical

# Fail closed if the header carries anything but the pinned value: a future
# script change (dropping SOURCE_DATE_EPOCH, or no longer reading `uname` from
# PATH) would otherwise silently put host data back into the binary.
pinned="$(echo "$(LC_TIME=C TZ=UTC date -d "@$SOURCE_DATE_EPOCH") $("$shims/uname")" | sed -e 's, *, ,g')"
grep -qxF "#define DATE \"$pinned\"" build/build.hpp || {
echo "ERROR: build/build.hpp DATE is not the pinned value:" >&2
grep '^#define DATE' build/build.hpp >&2 || true
exit 1
}

mkdir -p "$OUTPUT_DIR/usr/bin"
cp build/cadical "$OUTPUT_DIR/usr/bin/cadical"
26 changes: 25 additions & 1 deletion packages/lean/build.ncl
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,11 @@ let version = "4.35.0" in
# `bv_decide` shells out to the `cadical` SAT solver by name on PATH. An
# official Lean release bundles it in `bin/`; built from source we don't
# get one, and `bv_decide` fails with "could not execute external process
# 'cadical'" (found building aeneas-latest). Same release Lean pins.
# 'cadical'" (found building aeneas-latest). Our cadical is the latest
# release, not the older one Lean's CMake bundles: `bv_decide` verifies
# the solver's LRAT certificate, so a newer solver cannot make lean accept
# a false goal. The `bv_decide` test below is a smoke test of the pair.
# aeneas-latest's build is the real-goal coverage (see cadical/build.ncl).
cadical,
# `lean` and `libleanshared.so` link libssl.so.3 / libcrypto.so.3 (lake's
# HTTP client). Without this the binary does not START in a closure that
Expand Down Expand Up @@ -115,6 +119,26 @@ let version = "4.35.0" in
["/bin/bash", "-c", "mkdir -p /tmp/lk && cd /tmp/lk && printf 'import Lake\\nopen Lake DSL\\npackage t\\n@[default_target] lean_lib T\\n' > lakefile.lean && mkdir -p T && printf 'def hello := 42\\n' > T.lean && LAKE_NO_CACHE=1 HOME=/tmp/lk lake build"],
],
} | Test,
# `bv_decide` end to end, against the `cadical` runtime dep. Lean bit-blasts
# each goal to CNF, runs cadical (`--lrat --binary=true --quiet ...`), and
# replays the LRAT certificate in its own checker, so this proves both that
# cadical is found and that lean accepts the proofs this cadical writes.
# Each goal reaches the SAT solver (none is closed by bv_decide's
# preprocessing): with a fake `cadical` that prints garbage, every one of
# them fails. `csub` is a realistic query: Aeneas's ML-KEM conditional
# subtraction (aeneas-latest's Bvify.lean), restated on plain BitVec 32
# so it needs no Aeneas import. It takes under a second. The second
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low · "It takes under a second" states a measurement for this lean and cadical pair, but the author measured it on another pair. The reply on this PR says the time came from lean 4.34.1 with its bundled cadical 2.1.2, and that 3.0.1 was untested at that point. The claim matters because bv_decide has a solver timeout, so a reader can take this goal as safe from timeouts on 3.0.1. Measure it on lean 4.35.0 with cadical 3.0.1, or remove the sentence. This is one instance.

# command takes the SAT path: a false goal must be refuted with a
# counterexample, not proved and not errored.
bv_decide =
{
class = 'Standalone,
test_deps = [base],
cmds = [
["/bin/bash", "-c", "cd /tmp && printf '%s\\n' 'import Std.Tactic.BVDecide' 'theorem mul3 (x : BitVec 32) : x * 3 = x + x + x := by bv_decide' 'theorem andor (x y : BitVec 32) : (x &&& y) + (x ||| y) = x + y := by bv_decide' 'theorem mulcomm (x y : BitVec 8) : x * y = y * x := by bv_decide (config := { solverMode := .proof })' 'theorem csub (a b c2 c5 : BitVec 32) (ha : a < 3329#32) (hb : b < 3329#32) (hc2 : c2 = a + b - 3329#32) (hc5 : c5 = c2 + (3329#32 &&& (c2 >>> 16))) : c5 % 3329#32 = (a + b) % 3329#32 ∧ c5 < 3329#32 := by bv_decide' > bv.lean && lean bv.lean"],
["/bin/bash", "-c", "cd /tmp && printf '%s\\n' 'import Std.Tactic.BVDecide' 'theorem bad (x : BitVec 8) : x < x + 1 := by bv_decide' > cex.lean; out=\"$(lean cex.lean 2>&1)\"; echo \"$out\"; echo \"$out\" | grep -q 'The prover found a counterexample'"],
],
} | Test,
},
attrs =
{
Expand Down
Loading
0