8000 fix: kernel file locking replaces the PID sidecar (#317, #304) by adityamukho · Pull Request #332 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content

fix: kernel file locking replaces the PID sidecar (#317, #304) - #332

Merged
adityamukho merged 29 commits into
mainfrom
kernel-file-locking
Aug 22, 2026
Merged

adityamukho merged 29 commits into
mainfrom
kernel-file-locking

Conversation

@adityamukho
Copy link
Copy Markdown
Collaborator

Closes #317. Closes #304.

What was wrong

FileLock recorded the holder's PID in a .graph.lock sidecar and refused to open when the PID it read back equalled its own.

Every container's main process is PID 1 in its own namespace. So a container killed with SIGKILL never ran Drop, left the sidecar holding 1, and the replacement container — also PID 1 — read its own PID back, concluded a handle in this process held the lock, and refused. Permanently: no code path could clear it, and an operator had to delete the file by hand.

The pid == our_pid refusal was itself the fix for #304 (two FileBackends on one file in one process, each caching its own header.page_count, producing intermittent Page N out of bounds). The two defects were entangled — fixing one had created the other.

A PID cannot serve as a liveness token. It is not unique across PID namespaces and is recycled within one.

What this does

std::fs::File::try_lock on the .graph file itself. The kernel releases the lock when the process exits, however it exits, so a crashed holder leaves nothing behind. Because flock and OFD locks attach to the open file description rather than the process, a second open within one process is denied too — so #304 stays fixed with no PID bookkeeping at all, and the entanglement dissolves rather than being managed.

The sidecar is gone. A leftover from v1.2.x is ignored and never deleted, since a still-running old process may depend on it.

No new dependencies: File::try_lock is std (flock on Unix, LockFileEx on Windows). Binary size +1.49 KiB against CI's ~18 KiB headroom.

Why not PR #318

That PR proposed comparing /proc/<pid>/stat start times. Across PID namespaces /proc/<pid> does not refer to the holder at all, so a differing start time proves only "not the process I can see", never "dead" — converting a safe refusal into silent cross-process corruption. Full reasoning is on #318, which is closed with thanks; @ocasazza's report, reproduction, and diagnosis are what made this fix straightforward, and one of their tests is the ancestor of the regression test here.

Behaviour changes

  • MSRV 1.85 → 1.89. File::try_lock stabilised in 1.89. All 8 downstream repos install via dtolnay/rust-toolchain@stable and pin nothing, so the cascade is unaffected.
  • Windows: an open database can no longer be read by other processes. LockFileEx locks are mandatory, so copying an open .graph now fails where it previously succeeded. Unix stays advisory.
  • open() fails on filesystems that cannot lock, rather than proceeding unprotected. OpenOptions::allow_unlocked(true) opts out.
  • A cross-process-locked open reports its error ~375 ms later. fork transiently duplicates the lock-holding descriptor into a child until execve fires CLOEXEC, so any process spawning subprocesses — the normal pattern for the Python and Node bindings — could see a spurious WouldBlock. Bounded retry, 10 attempts, 5 ms doubling capped at 50 ms. SQLite solves the same problem with busy_timeout. Fail-closed survives; the same-process case never retries and stays instant.
  • A refused open on an unlockable filesystem may leave a 0-byte .graph — the file must exist before it can be locked.

OpenOptions gains a public field, which breaks struct-literal construction downstream. The release version is deliberately undecided and handled separately; the CHANGELOG's breaking-changes section carries the facts needed to choose, including that this is the first post-1.0 field addition and that #[non_exhaustive] would immunise future ones at the cost of being breaking itself.

Testing

1013 passed, 0 failed, 8 ignored across 34 binaries (was 999 passing; net +15).

The regression test reproduces #317 with unshare — two processes at PID 1 in separate namespaces sharing one file, no container runtime, about three seconds. Its catching power was verified, not assumed: run against pre-fix e056905 in a throwaway worktree, test_lock_survives_holder_death_in_another_pid_namespace fails with the #317 message while test_two_live_namespaced_holders_are_still_refused passes — the correct discrimination, since the old code did also refuse two live holders. That second test guards exactly what #318's mechanism would have broken.

Crash simulation moved from std::mem::forget to real aborting subprocesses. mem::forget stopped modelling a crash once the lock lived in the kernel: it leaks the File, so the descriptor stays open and the lock is held for the life of the test process. A test-only hook was rejected — #[doc(hidden)] pub would be callable in production, and #[cfg(test)] cannot reach tests/.

Known risk shipping unmitigated

If flock silently no-ops on some network filesystem, both processes receive Ok(()) and both write. Fail-closed does not cover it, because a silent no-op is not an error. Nightly NFS and Docker suites are designed to detect this and are deferred to #324, to be picked up on merge. The trade is still favourable: the sidecar this replaces could not evaluate a remote holder either, and bricked the database on any crash.

Follow-ups

#327 (extract duplicated crash harness), #328 (dangling doc link), #329 (stale "crash" test names), #330 (pin the ProceedUnlocked test-and-set), #331 (compat test descriptions). None blocking.

The wiki's Architecture.md has a matching commit, not yet pushed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

adityamukho and others added 22 commits August 22, 2026 10:29
Replaces the PID-sidecar FileLock with std::fs::File::try_lock. A PID is
not a liveness token: it is not unique across PID namespaces and is
recycled within one, so any PID-based liveness check is unsound wherever
containers run.

Supersedes PR #318, whose /proc start-time comparison cannot establish
liveness across namespaces and would convert a safe refusal into silent
cross-process corruption.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Scoped to the core fix plus the PID-namespace regression test. The nightly
NFS and Docker tiers are tracked separately in #324.

Also corrects two errors in the spec: OpenOptions exposes both public
fields and builder methods, so allow_unlocked needs both; and there are 14
crash-simulation call sites, not 23 -- the earlier figure counted comment
mentions of mem::forget alongside the actual calls.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Subagent-driven-development writes its ledger, briefs, and review packages
there. It is scratch, not source.
- Task 4's crash assertion required death by signal, which fails on
  Windows CI where abort() yields an exit code instead. wal_test.rs is
  gated only on not(wasm32), so it does run there.
- Task 5's Files block omitted examples/pid_ns_helper.rs, which its own
  Step 2 creates.
- Task 5's catching-power check used `git stash`, which cannot revert work
  already committed by Tasks 1-2 -- as written it would have tested the
  fixed code against itself and always passed. Uses a throwaway worktree
  at the pre-fix commit instead.
- Task 6 edited .wiki/, which does not exist in a worktree; it is a plain
  clone living only in the main checkout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Isolates the try_lock decision from the I/O so the unsupported-filesystem
branch is testable without a filesystem that cannot lock.

allow_unlocked deliberately does not apply to WouldBlock.
A PID is not a liveness token. It is not unique across PID namespaces and
is recycled within one, so the sidecar could never distinguish a crashed
holder from a live one. A container killed with SIGKILL left the sidecar
holding pid 1, and the replacement container -- also pid 1 -- refused to
open the database forever (#317).

File::try_lock puts the decision in the kernel, which releases the lock on
process death however it occurs. Because flock and OFD locks attach to the
open file description rather than the process, this also denies a second
open within one process, covering #304 with no bookkeeping.

A leftover sidecar from v1.2.x is ignored, never deleted: a still-running
old process may depend on it.
Three fixes to the kernel-file-lock work for #317/#304:

1. `_path_guard` is now declared before `file` in `FileBackend`. Rust drops
   struct fields in declaration order, so `file` (and the kernel lock it
   holds) was previously released before the registry entry was removed,
   opening a window for another thread to acquire the lock and insert its
   own entry that this guard's drop would then incorrectly delete.

2. `LockOutcome::ProceedUnlocked` now consults `already_open_here` and
   inserts into `OPEN_PATHS` before returning, matching the `Acquired` arm.
   On an unlockable filesystem there is no kernel lock, so the registry is
   the only thing preventing a second same-process handle to the same path
   from corrupting the file with two independent page tables (#304).

3. Repaired the `ProceedUnlocked` bail! message: its line-continuation
   backslashes had been dropped, leaving literal runs of spaces at each
   line join. Restored the `\`-continuation style used by the neighbouring
   bail! calls so it reads as one clean sentence, textually identical to
   the `LockOutcome::Held` same-process message.

`test_refused_open_does_not_corrupt_path_registry` was rewritten to assert
on the second refusal's error message rather than just its Err-ness.

Not done: an in-process test for "a refused open leaves drop-removal
intact" is not attempted here. That property is only observable with an
independent process holding the kernel lock (Task 5's subprocess
machinery) — inside one process, once the holder drops, try_lock simply
succeeds and the registry is never consulted.

Also not added: a test that reaches `ProceedUnlocked`'s same-process
refusal directly through `FileBackend::open_with`. That requires
`File::try_lock()` to return a real `TryLockError::Error` from an
actually unlockable filesystem (e.g. NFSv3 without lockd) — not
reproducible with a tempdir on a normal CI runner. The `classify` unit
tests already pin that `ProceedUnlocked` is produced correctly from
that raw `Result`; the registry-consulting behavior inside the
`open_with` match arm has no unlockable-filesystem-free way to exercise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Fails closed by default when a filesystem cannot lock. The escape hatch is
for single-writer-by-convention deployments on mounts without working
locks; it never bypasses a lock another handle holds.
Task 7 called for --all-targets, which lints test binaries carrying
several hundred pre-existing failures on main. CI runs --all-features.
Gating the branch on debt CI ignores would block it for unrelated
reasons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
- Gate FileBackend::open with #[cfg(test)] since production code now calls
  open_with directly. Add comment explaining this survives for test convenience.
  Fixes clippy dead-code warning that would fail CI.
- Rewrite checkpoint comment to remove stale 'manual lock deletion' example
  from .graph.lock sidecar era. Now accurately describes legitimate use cases:
  filesystems that cannot lock at all (NFSv3 without lockd, FUSE mounts).
.gitignore covered memory.graph but not its FTS index sidecars, so a
broad `git add` swept 72KB of local sqlite binaries into a commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
mem::forget no longer models a crash: it leaks the File, so the kernel
lock stays held for the life of the test process and no external cleanup
can release it.

A child that aborts is both the accurate model -- the kernel closes its
descriptors exactly as it would for a SIGKILLed container -- and the one
that needs no test-only hook in the shipping crate. #[cfg(test)] cannot
reach tests/, and #[doc(hidden)] pub would be callable in production.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Command::spawn forks the whole process before exec'ing the child; until
the child reaches execve and its close-on-exec descriptors close, it
transiently holds a duplicate of every flock its parent has open. A
concurrent try_lock() elsewhere -- another thread in this process, a
sibling process, or bindings that shell out (Python/Node commonly do)
-- can observe WouldBlock during that window even though no other
process is really contending for the file. This surfaced as flaky
"Database is locked by another process" failures on completely
unrelated tests once tests/wal_test.rs started spawning real crash
subprocesses.

SQLite solves the same class of problem with busy_timeout; this is our
version: retry try_lock() up to 10 times with backoff (5ms doubling,
capped at 50ms, ~375ms worst case) before concluding another process
genuinely holds the lock. The retry is skipped entirely when this
process already has the path open -- that conflict is unwinnable by
waiting, since we are the only thing that could release it -- so
test_second_open_in_same_process_is_refused stays instant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
…lock test

Review round 2 fixes:

1. test_explicit_tx_all_or_nothing_commit and
   wal_recover_committed_tx_crash_before_checkpoint had been converted to
   plain db.execute() calls in the crashing child, silently dropping the
   one thing they existed to pin: that WriteTransaction::commit() itself
   writes to the WAL before a crash, and that a multi-statement commit
   recovers atomically (all facts, not a subset). Neither property was
   asserted anywhere else in the tree. Added a CrashTx::{Implicit,Explicit}
   mode to run_crashing_child/crash_child_entrypoint (env var
   MINIGRAF_CRASH_TX) so the child can wrap statements in begin_write()/
   commit() instead of auto-committing each one, and restored both sites
   to CrashTx::Explicit.

2. The retry-on-WouldBlock fix in FileBackend::open_with had no test:
   nothing on the branch spawned a second process to actually hold the
   lock while another tried to open it, so the cross-process error
   message and the retry-budget bound were both unexercised -- and an
   already_open_here that wrongly returned true would have passed the
   whole suite. Added a lock-holding child (hold_lock_entrypoint) and a
   driver test that asserts: the open fails, it fails within one second
   (not by hanging out the full retry budget plus the holder's longer
   sleep), the error is the cross-process wording and not the
   same-process one, and a subsequent open succeeds once the holder
   exits.

Also: corrected the OPEN_PATHS doc comment (it now also gates the retry
decision, not just error wording) and the LOCK_RETRY_ATTEMPTS comment
that claimed a genuinely locked database "fails fast" (it pays the full
~375ms budget). Noted CRASH_CHILD_SPAWN_LOCK is now belt-and-braces now
that the real fix lives in open_with.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Two processes at PID 1 in separate namespaces sharing one file, via
unshare -- no container runtime needed. Verified to fail against the
pre-fix code: the killed holder's sidecar left pid 1 behind and the
replacement was refused permanently.

The second test guards the other direction: two live holders must still
be refused, which is what PR #318's start-time comparison would have
broken.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Review round 1 fix for #317's regression test.

A missing helper binary used to be a silent skip. Under any filtered
`cargo test` invocation that never triggers a rebuild, that reports a
green, vacuous pass with nothing in the output to say so. Now the skip
is reserved for the case that actually warrants it -- unshare_prefix()
returning None, i.e. this environment cannot create PID namespaces at
all. If namespaces work but the helper binary is absent, that can only
mean `cargo build --examples` was not run, so the test panics with a
message naming the binary instead of asserting nothing.

Also wrap container A's Child in a kill-on-drop guard in both tests.
std::process::Child does not kill its child on Drop, and both tests
hold container A across an assert! before their explicit kill/wait; if
that assert (or the wait_with_output().expect() further down in the
second test) panics first, the panic used to unwind straight past
cleanup and orphan container A -- PID 1 in its own namespace, still
mid-sleep, still holding the flock. The guard makes that unwind-safe
without restructuring either test.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
CHANGELOG carries the semver-relevant facts (new public option, MSRV
1.85 -> 1.89, behaviour change on unlockable filesystems, Windows
mandatory locks, bounded cross-process retry) so the release decision
can be made later.

Adds STG-025 through STG-027 with the actual error text; the lock
errors had no reference entries. Updates docs/TEST_COVERAGE.md,
CLAUDE.md, and README.md to the measured count (1021 tests: 1013
passing, 8 ignored) and reconciles the itemized breakdown -- including
the Wave 3 Conclusion section and the XTDB/Datomic compatibility
section headers, command-reference counts, and bullet lists (now one
accurate bullet per real test, matched in both directions against the
source), which had all drifted stale before this branch -- so no count
in the file contradicts another. Documents that hold_lock_entrypoint's
child process leaks a stray harness summary line into cargo test
output (easy to mistake for a second partial run of the lib unit test
binary), and gives a verified filter to exclude it when totalling.
Fixes a stale "Diagnostic only." comment on already_open_here in
src/storage/backend/file.rs -- its return value now also gates the
WouldBlock retry loop.
cargo fmt --check failed on three test files touched by the kernel
file locking work (migration_matrix_test.rs, pid_namespace_test.rs,
wal_test.rs) — formatting only, no behavior change.

Task 7 verification (docs/superpowers/plans/2026-08-22-kernel-file-locking.md).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Final-review fix wave, four items.

1. TOCTOU in the ProceedUnlocked arm. already_open_here() took OPEN_PATHS,
   read, and released it; the insert then took the lock again. Two threads
   opening the same path on a filesystem that cannot lock both observed
   "absent", both inserted, and both proceeded -- two FileBackends on one
   file with diverging page_count, which is exactly the #304 corruption.
   That arm has no kernel lock behind it, so the registry was the only
   guard, and its own comment said so while not providing it. Now one
   critical section using HashSet::insert's return value as test-and-set.
   A poisoned mutex fails closed.

2. Record the OpenOptions semver break in the CHANGELOG. Adding a public
   field to a struct that is not #[non_exhaustive] breaks struct-literal
   construction downstream. The release decision is deferred, so this
   documents the facts needed to make it: first post-1.0 field addition,
   the v0.19.0 precedent predates the guarantee, and #[non_exhaustive]
   would immunise future additions at the cost of being breaking itself.

3. Remove a flake window. The cross-process lock test held for 600ms
   against a ~375ms retry budget -- ~215ms of slack, across three CI
   platforms with 34 test binaries in flight. Now 3s, and the holder is
   killed rather than waited out, which is both faster than the original
   and tests release-on-SIGKILL instead of release-on-clean-exit.

4. Mark tiers 2 and 3 in the design spec as deferred to #324, and state
   plainly that this leaves the spec's own NFS silent-no-op risk
   unmitigated on merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
- The CHANGELOG said this branch updated "three" exhaustive OpenOptions
  literals. It updated two: the Default impl and one test. Only the test
  is a "downstream literal" in the sense the semver paragraph means.
  Verified by walking every OpenOptions literal in the tree.

- Two statements in the design spec still read as though Tier 2 ships
  with this branch: the prose at the NFS discussion, and the risk-register
  row, which a skim-reader may see without ever reaching the delivery
  note. Both now say plainly that the mitigation is deferred to #324 and
  the risk is unmitigated on merge -- and the risk row now notes why
  fail-closed does not cover it: a silent no-op returns Ok(()), so there
  is no error to fail on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Both were missed in the Task 6 doc sync. ROADMAP's release history
described v1.2.2's #304 fix with no note that this branch supersedes the
mechanism behind it, and CLAUDE.md's status line — which loads every
session — did not mention the change or the MSRV bump.

Neither states a version: the release decision is deferred, and both
point at the CHANGELOG's breaking-changes section instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
The #317 regression test passed locally and failed on GitHub's ubuntu
runner. Both tests ran (no skip), and the two-live-holders test passed,
so namespaces work there — the difference is how container A gets killed.

Command::kill() signals only the process we spawned. When `unshare` is
that process, `--kill-child` covers the rest: it sets PR_SET_PDEATHSIG on
the process that becomes PID 1. But GitHub's runners restrict unprivileged
user namespaces, so the probe selects the `sudo` fallback, and then a
privilege wrapper sits in between. SIGKILL runs no handler, so the wrapper
forwards nothing, `unshare` and its namespaced child survive, and the lock
is still held when container B tries to open.

That failure is maximally misleading: the #317 test fails identically
whether or not the bug is present, because the lock genuinely is still
held. A regression test that cannot distinguish those two states is worse
than no test.

The holder now leads its own process group and is killed as a group, with
the same privilege escalation the prefix used — root-owned descendants
reject an unprivileged signal with EPERM. The second test uses the same
path: its kill is cleanup only and cannot fail the assertion, but an
orphan there leaves a namespaced process on the flock for 300 seconds,
which on a shared runner breaks somebody else's test.

Both asserts now print the prefix actually selected and container B's
stdout and stderr, so the next failure of this kind names its own cause
instead of requiring a bisect against CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
@codecov
codecov Bot commented Aug 22, 2026
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 57.35294% with 29 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/storage/backend/file.rs 54.68% 29 Missing ⚠️

📢 Thoughts on this report? Let us know!

Attribution existed in the commit trailers, the design spec, and the PR
body, but not in the CHANGELOG — which is the one that ships to users and
gets read at release time.

They supplied the reproduction, identified the PID-namespace mechanism,
and traced the refusal back to the #304 fix that caused it. That analysis
is why this was quick to verify rather than quick to guess at.

The co-author trailer is repeated here deliberately. This repository
squashes with squash_merge_commit_message=COMMIT_MESSAGES, so the squash
body is every commit message concatenated in order, and git only parses
trailers in the FINAL paragraph. The trailers on a1bfec1 and 8cc1419 land
mid-body and would read as prose. Carrying it on the last commit puts it
where it will actually be parsed.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
adityamukho and others added 5 commits August 22, 2026 21:03
CI named the cause once the assertions printed it:

  prefix used: ["sudo", "-n", "unshare", "--pid", "--fork", "--mount-proc",
                "--kill-child"]
  container B stdout: OPEN_ERR Database is locked by another process

GitHub's runners restrict unprivileged user namespaces, so the probe takes
the sudo fallback. Ubuntu's sudo defaults to use_pty, which runs the
command in a NEW SESSION. Signalling the process we spawned reaches only
sudo, and SIGKILL runs no handler so nothing is forwarded; killing the
process group misses the holder too, because it is no longer in that
group. The previous attempt at a group kill failed for exactly this
reason.

Killing across that privilege boundary cannot be made reliable from here,
so stop trying. The holder now aborts itself: no unwinding, no Drop, no
checkpoint, no library-side lock release -- the same state a SIGKILLed
container leaves, which is what #317 is about. Test 1 asks for a zero
hold, so A dies the moment it holds the lock; test 2 asks for a bounded
2s hold, long enough for B to be refused while A lives and short enough
that a failure cannot strand a namespaced process on the flock.

KillOnDrop goes with it. It existed to stop a panic orphaning a holder
mid-way through a 300-second sleep; holders are now self-limiting, so it
guards nothing.

Catching power re-verified after the rework, not assumed: against pre-fix
e056905 the regression test still fails with the #317 message while the
two-live-holders test still passes -- the same discrimination as before.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
CI moved the failure from test 1 to test 2: the #317 regression test now
passes on the runner, and the two-live-holders test failed with container
B reporting OPEN_OK where it must be refused.

Cause was the fixed 2s hold. Container B has to start through
`sudo unshare` and attempt an open, and how long that takes differs
enormously between a laptop and a loaded runner. A hold that is generous
locally expires mid-attempt on CI, B then legitimately succeeds because
nothing is holding the lock any more, and the test reports that a live
holder was not excluded — which was never what happened.

Any fixed number is a guess at someone else's machine. The holder now
waits for a release marker the test writes once B's result is recorded,
so the hold is exactly as long as it needs to be. The helper keeps a 60s
deadline purely as a backstop, so a panic in the parent cannot strand a
namespaced process on the flock.

Faster too: 0.64s against 2.16s, because nothing waits out a sleep that
was mostly slack.

Catching power re-verified against pre-fix e056905 with a freshly built
helper -- the regression test fails there with the #317 message while the
two-live-holders test passes.

Note for anyone iterating on this: `cargo build --tests` and
`cargo test --test <name>` do NOT rebuild `examples/`, so the helper can
go stale and silently change behaviour under you. Plain `cargo test`,
which is what CI runs, does rebuild it.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
The merge brought #325 and #326, adding two lib unit tests, so every count
this branch had just synced was stale again: 1015 passing, 8 ignored,
1023 total, with 678 in the lib binary.

Updated CLAUDE.md, README.md, and docs/TEST_COVERAGE.md, including the
itemized Test Summary — its 33 bullets sum to exactly 1023, verified
rather than eyeballed.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
sudo defaults to env_reset: everything outside a small allowlist is
stripped before the command runs. GitHub's runners restrict unprivileged
user namespaces, so the test takes the `sudo unshare ...` fallback, and
the holder never saw MINIGRAF_NS_RELEASE_MARKER at all. It took the
"nothing to wait for" path and aborted the instant it held the lock, so
container B found the file free and opened it — and the test reported
that a live holder had not been excluded, when there had been no live
holder to exclude.

This also corrects the previous commit's diagnosis. That one blamed a
fixed 2s hold expiring on a slow runner. The hold was never the problem:
MINIGRAF_NS_HOLD_MILLIS was stripped by the same mechanism, so A aborted
immediately in that version too. The release marker was the right change
for a different reason — it removes the guess about someone else's
machine — but the failure it was meant to fix had another cause entirely.

Arguments cross the sudo boundary; environment variables do not.

Verified locally under `env -i`, which reproduces env_reset exactly: the
holder stays alive with a completely stripped environment and aborts only
once the marker appears. That is the CI condition, reproduced on a machine
with no passwordless sudo.

Catching power re-checked against pre-fix e056905: the regression test
still fails there with the #317 message, the two-live-holders test still
passes.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
Windows CI failed with os error 33, "another process has locked a portion
of the file", on a read whose "other process" was the test itself.

Windows file locks are mandatory, not advisory, and they exclude every
handle but the one holding them — a second handle in the same process
included. test_manual_checkpoint_deletes_wal opened the .graph file
through a fresh `File` to assert the header reflected the checkpoint,
while the database was still open and holding the lock. On Unix the lock
is advisory and that read succeeds, which is why this only ever failed on
one leg of the matrix.

Moving the read after the existing `drop(db)` fixes it and asserts the
same thing. Two other raw reads in the suite already closed their handle
first, so this was the only site.

Also corrects the CHANGELOG, which claimed only that "other processes"
cannot read an open database on Windows. That understated it: the
exclusion covers your own process's other handles, which is the case a
user is far more likely to hit — inspecting a file they themselves have
open. The wiki carried the same understatement and is corrected in its
own repository.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf
@adityamukho
adityamukho merged commit 5450e94 into main Aug 22, 2026
16 of 17 checks passed
@adityamukho
adityamukho deleted the kernel-file-locking branch August 22, 2026 17:05
adityamukho added a commit that referenced this pull request Aug 23, 2026
* ci: add nightly NFS and Docker lock tests (#324)

Tiers 2 and 3 from the kernel file-locking design, deferred from #317/#332:
the per-PR suite covers PID namespaces but cannot reach how
std::fs::File::try_lock behaves on a network filesystem or in real
containers.

tests/nfs_lock_test.rs asserts the core #324 property on a loopback NFSv4
mount and an NFSv3 mount without lockd: a lock held by one process must
never let a second process also believe it holds it. It also proves
allow_unlocked is consulted only on a mount that cannot lock at all, never
to bypass a live lock. tests/docker_lock_test.rs runs the same #317
regression shape (a killed holder's replacement must still be able to
open) across two containers sharing a Docker volume, matching the
original report literally.

Both reuse examples/pid_ns_helper.rs, which gains an explicit mode match
(an unrecognized mode now fails loudly instead of silently behaving like
"open") and a new open-unlocked mode.

Locally verifiable pieces (mode validation, open-unlocked on an ordinary
filesystem, docker_available() reflecting a sandbox with no Docker) are
covered by TDD here. The NFS- and Docker-gated assertions need root and a
container runtime this sandbox doesn't have; they skip cleanly without
their environment and are exercised for real by the new
.github/workflows/nfs-lock.yml nightly workflow, which also records
failures as tracking issues, matching fuzz.yml/property-tests.yml.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU

* fix(test): remove environment-specific docker_available() assertion

test_docker_availability_check_reflects_this_sandbox asserted docker was
never reachable, true only in the local sandbox this branch was written
in. GitHub's ubuntu-latest runners have Docker preinstalled and running,
so the assertion failed there -- while, in the same run, the two real
gated tests it was meant to backstop actually exercised Docker for real
and passed. Those two are the genuine proof; this one just encoded an
environment assumption as a hard failure.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU

* fix(ci): build examples before tarpaulin so lock tests find the helper

cargo tarpaulin does not build examples/ before running tests, unlike
plain cargo test. tests/nfs_lock_test.rs and tests/docker_lock_test.rs
resolve examples/pid_ns_helper relative to the test binary's own target
dir, so it must already be built. tests/pid_namespace_test.rs never hit
this because it skips before reaching that check when unshare isn't
available in tarpaulin's environment; the new tests don't have an
equivalent early-out.

Reproduced and verified the fix locally with cargo tarpaulin directly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU

* fix(ci): give lock tests an env-var override for the helper path

cargo tarpaulin deletes target/debug/examples/ as part of its own
instrumented build, regardless of when it was built -- a plain
`cargo build --examples` step before the tarpaulin run doesn't survive
it. Reproduced locally against cargo-tarpaulin 0.35.2 from a clean
checkout: the previous fix (build-examples-first) still failed with
"helper binary not found", and target/debug/examples/ was empty after
tarpaulin ran.

nfs_lock_test.rs and docker_lock_test.rs now check
MINIGRAF_PID_NS_HELPER before falling back to the current_exe()-derived
path. coverage.yml builds the helper into an isolated --target-dir
tarpaulin never touches and points the variable at it. Verified against
a clean checkout with cargo tarpaulin directly: both suites pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YMWSutMLJJ7Y5b8hs9ZaCU

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant

0