Repository navigation
fix: kernel file locking replaces the PID sidecar (#317, #304) - #332
Merged
Merged
Conversation
Replaces the PID-sidecar FileLock with std::fs::File::try_lock. A PID is not a liveness token: it is not unique across PID namespaces and is recycled within one, so any PID-based liveness check is unsound wherever containers run. Supersedes PR #318, whose /proc start-time comparison cannot establish liveness across namespaces and would convert a safe refusal into silent cross-process corruption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Scoped to the core fix plus the PID-namespace regression test. The nightly NFS and Docker tiers are tracked separately in #324. Also corrects two errors in the spec: OpenOptions exposes both public fields and builder methods, so allow_unlocked needs both; and there are 14 crash-simulation call sites, not 23 -- the earlier figure counted comment mentions of mem::forget alongside the actual calls. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Subagent-driven-development writes its ledger, briefs, and review packages there. It is scratch, not source.
- Task 4's crash assertion required death by signal, which fails on Windows CI where abort() yields an exit code instead. wal_test.rs is gated only on not(wasm32), so it does run there. - Task 5's Files block omitted examples/pid_ns_helper.rs, which its own Step 2 creates. - Task 5's catching-power check used `git stash`, which cannot revert work already committed by Tasks 1-2 -- as written it would have tested the fixed code against itself and always passed. Uses a throwaway worktree at the pre-fix commit instead. - Task 6 edited .wiki/, which does not exist in a worktree; it is a plain clone living only in the main checkout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Isolates the try_lock decision from the I/O so the unsupported-filesystem branch is testable without a filesystem that cannot lock. allow_unlocked deliberately does not apply to WouldBlock.
A PID is not a liveness token. It is not unique across PID namespaces and is recycled within one, so the sidecar could never distinguish a crashed holder from a live one. A container killed with SIGKILL left the sidecar holding pid 1, and the replacement container -- also pid 1 -- refused to open the database forever (#317). File::try_lock puts the decision in the kernel, which releases the lock on process death however it occurs. Because flock and OFD locks attach to the open file description rather than the process, this also denies a second open within one process, covering #304 with no bookkeeping. A leftover sidecar from v1.2.x is ignored, never deleted: a still-running old process may depend on it.
Three fixes to the kernel-file-lock work for #317/#304: 1. `_path_guard` is now declared before `file` in `FileBackend`. Rust drops struct fields in declaration order, so `file` (and the kernel lock it holds) was previously released before the registry entry was removed, opening a window for another thread to acquire the lock and insert its own entry that this guard's drop would then incorrectly delete. 2. `LockOutcome::ProceedUnlocked` now consults `already_open_here` and inserts into `OPEN_PATHS` before returning, matching the `Acquired` arm. On an unlockable filesystem there is no kernel lock, so the registry is the only thing preventing a second same-process handle to the same path from corrupting the file with two independent page tables (#304). 3. Repaired the `ProceedUnlocked` bail! message: its line-continuation backslashes had been dropped, leaving literal runs of spaces at each line join. Restored the `\`-continuation style used by the neighbouring bail! calls so it reads as one clean sentence, textually identical to the `LockOutcome::Held` same-process message. `test_refused_open_does_not_corrupt_path_registry` was rewritten to assert on the second refusal's error message rather than just its Err-ness. Not done: an in-process test for "a refused open leaves drop-removal intact" is not attempted here. That property is only observable with an independent process holding the kernel lock (Task 5's subprocess machinery) — inside one process, once the holder drops, try_lock simply succeeds and the registry is never consulted. Also not added: a test that reaches `ProceedUnlocked`'s same-process refusal directly through `FileBackend::open_with`. That requires `File::try_lock()` to return a real `TryLockError::Error` from an actually unlockable filesystem (e.g. NFSv3 without lockd) — not reproducible with a tempdir on a normal CI runner. The `classify` unit tests already pin that `ProceedUnlocked` is produced correctly from that raw `Result`; the registry-consulting behavior inside the `open_with` match arm has no unlockable-filesystem-free way to exercise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Fails closed by default when a filesystem cannot lock. The escape hatch is for single-writer-by-convention deployments on mounts without working locks; it never bypasses a lock another handle holds.
Task 7 called for --all-targets, which lints test binaries carrying several hundred pre-existing failures on main. CI runs --all-features. Gating the branch on debt CI ignores would block it for unrelated reasons. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
- Gate FileBackend::open with #[cfg(test)] since production code now calls open_with directly. Add comment explaining this survives for test convenience. Fixes clippy dead-code warning that would fail CI. - Rewrite checkpoint comment to remove stale 'manual lock deletion' example from .graph.lock sidecar era. Now accurately describes legitimate use cases: filesystems that cannot lock at all (NFSv3 without lockd, FUSE mounts).
.gitignore covered memory.graph but not its FTS index sidecars, so a broad `git add` swept 72KB of local sqlite binaries into a commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
mem::forget no longer models a crash: it leaks the File, so the kernel lock stays held for the life of the test process and no external cleanup can release it. A child that aborts is both the accurate model -- the kernel closes its descriptors exactly as it would for a SIGKILLed container -- and the one that needs no test-only hook in the shipping crate. #[cfg(test)] cannot reach tests/, and #[doc(hidden)] pub would be callable in production. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Command::spawn forks the whole process before exec'ing the child; until the child reaches execve and its close-on-exec descriptors close, it transiently holds a duplicate of every flock its parent has open. A concurrent try_lock() elsewhere -- another thread in this process, a sibling process, or bindings that shell out (Python/Node commonly do) -- can observe WouldBlock during that window even though no other process is really contending for the file. This surfaced as flaky "Database is locked by another process" failures on completely unrelated tests once tests/wal_test.rs started spawning real crash subprocesses. SQLite solves the same class of problem with busy_timeout; this is our version: retry try_lock() up to 10 times with backoff (5ms doubling, capped at 50ms, ~375ms worst case) before concluding another process genuinely holds the lock. The retry is skipped entirely when this process already has the path open -- that conflict is unwinnable by waiting, since we are the only thing that could release it -- so test_second_open_in_same_process_is_refused stays instant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
…lock test
Review round 2 fixes:
1. test_explicit_tx_all_or_nothing_commit and
wal_recover_committed_tx_crash_before_checkpoint had been converted to
plain db.execute() calls in the crashing child, silently dropping the
one thing they existed to pin: that WriteTransaction::commit() itself
writes to the WAL before a crash, and that a multi-statement commit
recovers atomically (all facts, not a subset). Neither property was
asserted anywhere else in the tree. Added a CrashTx::{Implicit,Explicit}
mode to run_crashing_child/crash_child_entrypoint (env var
MINIGRAF_CRASH_TX) so the child can wrap statements in begin_write()/
commit() instead of auto-committing each one, and restored both sites
to CrashTx::Explicit.
2. The retry-on-WouldBlock fix in FileBackend::open_with had no test:
nothing on the branch spawned a second process to actually hold the
lock while another tried to open it, so the cross-process error
message and the retry-budget bound were both unexercised -- and an
already_open_here that wrongly returned true would have passed the
whole suite. Added a lock-holding child (hold_lock_entrypoint) and a
driver test that asserts: the open fails, it fails within one second
(not by hanging out the full retry budget plus the holder's longer
sleep), the error is the cross-process wording and not the
same-process one, and a subsequent open succeeds once the holder
exits.
Also: corrected the OPEN_PATHS doc comment (it now also gates the retry
decision, not just error wording) and the LOCK_RETRY_ATTEMPTS comment
that claimed a genuinely locked database "fails fast" (it pays the full
~375ms budget). Noted CRASH_CHILD_SPAWN_LOCK is now belt-and-braces now
that the real fix lives in open_with.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Two processes at PID 1 in separate namespaces sharing one file, via unshare -- no container runtime needed. Verified to fail against the pre-fix code: the killed holder's sidecar left pid 1 behind and the replacement was refused permanently. The second test guards the other direction: two live holders must still be refused, which is what PR #318's start-time comparison would have broken. Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Review round 1 fix for #317's regression test. A missing helper binary used to be a silent skip. Under any filtered `cargo test` invocation that never triggers a rebuild, that reports a green, vacuous pass with nothing in the output to say so. Now the skip is reserved for the case that actually warrants it -- unshare_prefix() returning None, i.e. this environment cannot create PID namespaces at all. If namespaces work but the helper binary is absent, that can only mean `cargo build --examples` was not run, so the test panics with a message naming the binary instead of asserting nothing. Also wrap container A's Child in a kill-on-drop guard in both tests. std::process::Child does not kill its child on Drop, and both tests hold container A across an assert! before their explicit kill/wait; if that assert (or the wait_with_output().expect() further down in the second test) panics first, the panic used to unwind straight past cleanup and orphan container A -- PID 1 in its own namespace, still mid-sleep, still holding the flock. The guard makes that unwind-safe without restructuring either test. Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
CHANGELOG carries the semver-relevant facts (new public option, MSRV 1.85 -> 1.89, behaviour change on unlockable filesystems, Windows mandatory locks, bounded cross-process retry) so the release decision can be made later. Adds STG-025 through STG-027 with the actual error text; the lock errors had no reference entries. Updates docs/TEST_COVERAGE.md, CLAUDE.md, and README.md to the measured count (1021 tests: 1013 passing, 8 ignored) and reconciles the itemized breakdown -- including the Wave 3 Conclusion section and the XTDB/Datomic compatibility section headers, command-reference counts, and bullet lists (now one accurate bullet per real test, matched in both directions against the source), which had all drifted stale before this branch -- so no count in the file contradicts another. Documents that hold_lock_entrypoint's child process leaks a stray harness summary line into cargo test output (easy to mistake for a second partial run of the lib unit test binary), and gives a verified filter to exclude it when totalling. Fixes a stale "Diagnostic only." comment on already_open_here in src/storage/backend/file.rs -- its return value now also gates the WouldBlock retry loop.
cargo fmt --check failed on three test files touched by the kernel file locking work (migration_matrix_test.rs, pid_namespace_test.rs, wal_test.rs) — formatting only, no behavior change. Task 7 verification (docs/superpowers/plans/2026-08-22-kernel-file-locking.md). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Final-review fix wave, four items. 1. TOCTOU in the ProceedUnlocked arm. already_open_here() took OPEN_PATHS, read, and released it; the insert then took the lock again. Two threads opening the same path on a filesystem that cannot lock both observed "absent", both inserted, and both proceeded -- two FileBackends on one file with diverging page_count, which is exactly the #304 corruption. That arm has no kernel lock behind it, so the registry was the only guard, and its own comment said so while not providing it. Now one critical section using HashSet::insert's return value as test-and-set. A poisoned mutex fails closed. 2. Record the OpenOptions semver break in the CHANGELOG. Adding a public field to a struct that is not #[non_exhaustive] breaks struct-literal construction downstream. The release decision is deferred, so this documents the facts needed to make it: first post-1.0 field addition, the v0.19.0 precedent predates the guarantee, and #[non_exhaustive] would immunise future additions at the cost of being breaking itself. 3. Remove a flake window. The cross-process lock test held for 600ms against a ~375ms retry budget -- ~215ms of slack, across three CI platforms with 34 test binaries in flight. Now 3s, and the holder is killed rather than waited out, which is both faster than the original and tests release-on-SIGKILL instead of release-on-clean-exit. 4. Mark tiers 2 and 3 in the design spec as deferred to #324, and state plainly that this leaves the spec's own NFS silent-no-op risk unmitigated on merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
- The CHANGELOG said this branch updated "three" exhaustive OpenOptions literals. It updated two: the Default impl and one test. Only the test is a "downstream literal" in the sense the semver paragraph means. Verified by walking every OpenOptions literal in the tree. - Two statements in the design spec still read as though Tier 2 ships with this branch: the prose at the NFS discussion, and the risk-register row, which a skim-reader may see without ever reaching the delivery note. Both now say plainly that the mitigation is deferred to #324 and the risk is unmitigated on merge -- and the risk row now notes why fail-closed does not cover it: a silent no-op returns Ok(()), so there is no error to fail on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Both were missed in the Task 6 doc sync. ROADMAP's release history described v1.2.2's #304 fix with no note that this branch supersedes the mechanism behind it, and CLAUDE.md's status line — which loads every session — did not mention the change or the MSRV bump. Neither states a version: the release decision is deferred, and both point at the CHANGELOG's breaking-changes section instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
The #317 regression test passed locally and failed on GitHub's ubuntu runner. Both tests ran (no skip), and the two-live-holders test passed, so namespaces work there — the difference is how container A gets killed. Command::kill() signals only the process we spawned. When `unshare` is that process, `--kill-child` covers the rest: it sets PR_SET_PDEATHSIG on the process that becomes PID 1. But GitHub's runners restrict unprivileged user namespaces, so the probe selects the `sudo` fallback, and then a privilege wrapper sits in between. SIGKILL runs no handler, so the wrapper forwards nothing, `unshare` and its namespaced child survive, and the lock is still held when container B tries to open. That failure is maximally misleading: the #317 test fails identically whether or not the bug is present, because the lock genuinely is still held. A regression test that cannot distinguish those two states is worse than no test. The holder now leads its own process group and is killed as a group, with the same privilege escalation the prefix used — root-owned descendants reject an unprivileged signal with EPERM. The second test uses the same path: its kill is cleanup only and cannot fail the assertion, but an orphan there leaves a namespaced process on the flock for 300 seconds, which on a shared runner breaks somebody else's test. Both asserts now print the prefix actually selected and container B's stdout and stderr, so the next failure of this kind names its own cause instead of requiring a bisect against CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Attribution existed in the commit trailers, the design spec, and the PR body, but not in the CHANGELOG — which is the one that ships to users and gets read at release time. They supplied the reproduction, identified the PID-namespace mechanism, and traced the refusal back to the #304 fix that caused it. That analysis is why this was quick to verify rather than quick to guess at. The co-author trailer is repeated here deliberately. This repository squashes with squash_merge_commit_message=COMMIT_MESSAGES, so the squash body is every commit message concatenated in order, and git only parses trailers in the FINAL paragraph. The trailers on a1bfec1 and 8cc1419 land mid-body and would read as prose. Carrying it on the last commit puts it where it will actually be parsed. Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
adityamukho
force-pushed
the
kernel-file-locking
branch
from
August 22, 2026 15:28
38e6cc7 to
e4d90b1
Compare
CI named the cause once the assertions printed it:
prefix used: ["sudo", "-n", "unshare", "--pid", "--fork", "--mount-proc",
"--kill-child"]
container B stdout: OPEN_ERR Database is locked by another process
GitHub's runners restrict unprivileged user namespaces, so the probe takes
the sudo fallback. Ubuntu's sudo defaults to use_pty, which runs the
command in a NEW SESSION. Signalling the process we spawned reaches only
sudo, and SIGKILL runs no handler so nothing is forwarded; killing the
process group misses the holder too, because it is no longer in that
group. The previous attempt at a group kill failed for exactly this
reason.
Killing across that privilege boundary cannot be made reliable from here,
so stop trying. The holder now aborts itself: no unwinding, no Drop, no
checkpoint, no library-side lock release -- the same state a SIGKILLed
container leaves, which is what #317 is about. Test 1 asks for a zero
hold, so A dies the moment it holds the lock; test 2 asks for a bounded
2s hold, long enough for B to be refused while A lives and short enough
that a failure cannot strand a namespaced process on the flock.
KillOnDrop goes with it. It existed to stop a panic orphaning a holder
mid-way through a 300-second sleep; holders are now self-limiting, so it
guards nothing.
Catching power re-verified after the rework, not assumed: against pre-fix
e056905 the regression test still fails with the #317 message while the
two-live-holders test still passes -- the same discrimination as before.
Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
CI moved the failure from test 1 to test 2: the #317 regression test now passes on the runner, and the two-live-holders test failed with container B reporting OPEN_OK where it must be refused. Cause was the fixed 2s hold. Container B has to start through `sudo unshare` and attempt an open, and how long that takes differs enormously between a laptop and a loaded runner. A hold that is generous locally expires mid-attempt on CI, B then legitimately succeeds because nothing is holding the lock any more, and the test reports that a live holder was not excluded — which was never what happened. Any fixed number is a guess at someone else's machine. The holder now waits for a release marker the test writes once B's result is recorded, so the hold is exactly as long as it needs to be. The helper keeps a 60s deadline purely as a backstop, so a panic in the parent cannot strand a namespaced process on the flock. Faster too: 0.64s against 2.16s, because nothing waits out a sleep that was mostly slack. Catching power re-verified against pre-fix e056905 with a freshly built helper -- the regression test fails there with the #317 message while the two-live-holders test passes. Note for anyone iterating on this: `cargo build --tests` and `cargo test --test <name>` do NOT rebuild `examples/`, so the helper can go stale and silently change behaviour under you. Plain `cargo test`, which is what CI runs, does rebuild it. Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
The merge brought #325 and #326, adding two lib unit tests, so every count this branch had just synced was stale again: 1015 passing, 8 ignored, 1023 total, with 678 in the lib binary. Updated CLAUDE.md, README.md, and docs/TEST_COVERAGE.md, including the itemized Test Summary — its 33 bullets sum to exactly 1023, verified rather than eyeballed. Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
sudo defaults to env_reset: everything outside a small allowlist is stripped before the command runs. GitHub's runners restrict unprivileged user namespaces, so the test takes the `sudo unshare ...` fallback, and the holder never saw MINIGRAF_NS_RELEASE_MARKER at all. It took the "nothing to wait for" path and aborted the instant it held the lock, so container B found the file free and opened it — and the test reported that a live holder had not been excluded, when there had been no live holder to exclude. This also corrects the previous commit's diagnosis. That one blamed a fixed 2s hold expiring on a slow runner. The hold was never the problem: MINIGRAF_NS_HOLD_MILLIS was stripped by the same mechanism, so A aborted immediately in that version too. The release marker was the right change for a different reason — it removes the guess about someone else's machine — but the failure it was meant to fix had another cause entirely. Arguments cross the sudo boundary; environment variables do not. Verified locally under `env -i`, which reproduces env_reset exactly: the holder stays alive with a completely stripped environment and aborts only once the marker appears. That is the CI condition, reproduced on a machine with no passwordless sudo. Catching power re-checked against pre-fix e056905: the regression test still fails there with the #317 message, the two-live-holders test still passes. Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Windows CI failed with os error 33, "another process has locked a portion of the file", on a read whose "other process" was the test itself. Windows file locks are mandatory, not advisory, and they exclude every handle but the one holding them — a second handle in the same process included. test_manual_checkpoint_deletes_wal opened the .graph file through a fresh `File` to assert the header reflected the checkpoint, while the database was still open and holding the lock. On Unix the lock is advisory and that read succeeds, which is why this only ever failed on one leg of the matrix. Moving the read after the existing `drop(db)` fixes it and asserts the same thing. Two other raw reads in the suite already closed their handle first, so this was the only site. Also corrects the CHANGELOG, which claimed only that "other processes" cannot read an open database on Windows. That understated it: the exclusion covers your own process's other handles, which is the case a user is far more likely to hit — inspecting a file they themselves have open. The wiki carried the same understatement and is corrected in its own repository. Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
4 of 5 tasks
adityamukho
added a commit
that referenced
this pull request
Aug 23, 2026
* ci: add nightly NFS and Docker lock tests (#324) Tiers 2 and 3 from the kernel file-locking design, deferred from #317/#332: the per-PR suite covers PID namespaces but cannot reach how std::fs::File::try_lock behaves on a network filesystem or in real containers. tests/nfs_lock_test.rs asserts the core #324 property on a loopback NFSv4 mount and an NFSv3 mount without lockd: a lock held by one process must never let a second process also believe it holds it. It also proves allow_unlocked is consulted only on a mount that cannot lock at all, never to bypass a live lock. tests/docker_lock_test.rs runs the same #317 regression shape (a killed holder's replacement must still be able to open) across two containers sharing a Docker volume, matching the original report literally. Both reuse examples/pid_ns_helper.rs, which gains an explicit mode match (an unrecognized mode now fails loudly instead of silently behaving like "open") and a new open-unlocked mode. Locally verifiable pieces (mode validation, open-unlocked on an ordinary filesystem, docker_available() reflecting a sandbox with no Docker) are covered by TDD here. The NFS- and Docker-gated assertions need root and a container runtime this sandbox doesn't have; they skip cleanly without their environment and are exercised for real by the new .github/workflows/nfs-lock.yml nightly workflow, which also records failures as tracking issues, matching fuzz.yml/property-tests.yml. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU * fix(test): remove environment-specific docker_available() assertion test_docker_availability_check_reflects_this_sandbox asserted docker was never reachable, true only in the local sandbox this branch was written in. GitHub's ubuntu-latest runners have Docker preinstalled and running, so the assertion failed there -- while, in the same run, the two real gated tests it was meant to backstop actually exercised Docker for real and passed. Those two are the genuine proof; this one just encoded an environment assumption as a hard failure. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU * fix(ci): build examples before tarpaulin so lock tests find the helper cargo tarpaulin does not build examples/ before running tests, unlike plain cargo test. tests/nfs_lock_test.rs and tests/docker_lock_test.rs resolve examples/pid_ns_helper relative to the test binary's own target dir, so it must already be built. tests/pid_namespace_test.rs never hit this because it skips before reaching that check when unshare isn't available in tarpaulin's environment; the new tests don't have an equivalent early-out. Reproduced and verified the fix locally with cargo tarpaulin directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU * fix(ci): give lock tests an env-var override for the helper path cargo tarpaulin deletes target/debug/examples/ as part of its own instrumented build, regardless of when it was built -- a plain `cargo build --examples` step before the tarpaulin run doesn't survive it. Reproduced locally against cargo-tarpaulin 0.35.2 from a clean checkout: the previous fix (build-examples-first) still failed with "helper binary not found", and target/debug/examples/ was empty after tarpaulin ran. nfs_lock_test.rs and docker_lock_test.rs now check MINIGRAF_PID_NS_HELPER before falling back to the current_exe()-derived path. coverage.yml builds the helper into an isolated --target-dir tarpaulin never touches and points the variable at it. Verified against a clean checkout with cargo tarpaulin directly: both suites pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #317. Closes #304.
What was wrong
FileLockrecorded the holder's PID in a.graph.locksidecar and refused to open when the PID it read back equalled its own.Every container's main process is PID 1 in its own namespace. So a container killed with SIGKILL never ran
Drop, left the sidecar holding1, and the replacement container — also PID 1 — read its own PID back, concluded a handle in this process held the lock, and refused. Permanently: no code path could clear it, and an operator had to delete the file by hand.The
pid == our_pidrefusal was itself the fix for #304 (twoFileBackends on one file in one process, each caching its ownheader.page_count, producing intermittentPage N out of bounds). The two defects were entangled — fixing one had created the other.A PID cannot serve as a liveness token. It is not unique across PID namespaces and is recycled within one.
What this does
std::fs::File::try_lockon the.graphfile itself. The kernel releases the lock when the process exits, however it exits, so a crashed holder leaves nothing behind. Becauseflockand OFD locks attach to the open file description rather than the process, a second open within one process is denied too — so #304 stays fixed with no PID bookkeeping at all, and the entanglement dissolves rather than being managed.The sidecar is gone. A leftover from v1.2.x is ignored and never deleted, since a still-running old process may depend on it.
No new dependencies:
File::try_lockis std (flockon Unix,LockFileExon Windows). Binary size +1.49 KiB against CI's ~18 KiB headroom.Why not PR #318
That PR proposed comparing
/proc/<pid>/statstart times. Across PID namespaces/proc/<pid>does not refer to the holder at all, so a differing start time proves only "not the process I can see", never "dead" — converting a safe refusal into silent cross-process corruption. Full reasoning is on #318, which is closed with thanks; @ocasazza's report, reproduction, and diagnosis are what made this fix straightforward, and one of their tests is the ancestor of the regression test here.Behaviour changes
File::try_lockstabilised in 1.89. All 8 downstream repos install viadtolnay/rust-toolchain@stableand pin nothing, so the cascade is unaffected.LockFileExlocks are mandatory, so copying an open.graphnow fails where it previously succeeded. Unix stays advisory.open()fails on filesystems that cannot lock, rather than proceeding unprotected.OpenOptions::allow_unlocked(true)opts out.forktransiently duplicates the lock-holding descriptor into a child untilexecvefires CLOEXEC, so any process spawning subprocesses — the normal pattern for the Python and Node bindings — could see a spuriousWouldBlock. Bounded retry, 10 attempts, 5 ms doubling capped at 50 ms. SQLite solves the same problem withbusy_timeout. Fail-closed survives; the same-process case never retries and stays instant..graph— the file must exist before it can be locked.OpenOptionsgains a public field, which breaks struct-literal construction downstream. The release version is deliberately undecided and handled separately; the CHANGELOG's breaking-changes section carries the facts needed to choose, including that this is the first post-1.0 field addition and that#[non_exhaustive]would immunise future ones at the cost of being breaking itself.Testing
1013 passed, 0 failed, 8 ignored across 34 binaries (was 999 passing; net +15).
The regression test reproduces #317 with
unshare— two processes at PID 1 in separate namespaces sharing one file, no container runtime, about three seconds. Its catching power was verified, not assumed: run against pre-fixe056905in a throwaway worktree,test_lock_survives_holder_death_in_another_pid_namespacefails with the #317 message whiletest_two_live_namespaced_holders_are_still_refusedpasses — the correct discrimination, since the old code did also refuse two live holders. That second test guards exactly what #318's mechanism would have broken.Crash simulation moved from
std::mem::forgetto real aborting subprocesses.mem::forgetstopped modelling a crash once the lock lived in the kernel: it leaks theFile, so the descriptor stays open and the lock is held for the life of the test process. A test-only hook was rejected —#[doc(hidden)] pubwould be callable in production, and#[cfg(test)]cannot reachtests/.Known risk shipping unmitigated
If
flocksilently no-ops on some network filesystem, both processes receiveOk(())and both write. Fail-closed does not cover it, because a silent no-op is not an error. Nightly NFS and Docker suites are designed to detect this and are deferred to #324, to be picked up on merge. The trade is still favourable: the sidecar this replaces could not evaluate a remote holder either, and bricked the database on any crash.Follow-ups
#327 (extract duplicated crash harness), #328 (dangling doc link), #329 (stale "crash" test names), #330 (pin the ProceedUnlocked test-and-set), #331 (compat test descriptions). None blocking.
The wiki's
Architecture.mdhas a matching commit, not yet pushed.🤖 Generated with Claude Code
https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf