8000 fix(storage): reclaim stale lock when PID aliases across PID namespaces by ocasazza · Pull Request #318 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content

fix(storage): reclaim stale lock when PID aliases across PID namespaces - #318

Closed
ocasazza wants to merge 2 commits into
project-minigraf:mainfrom
ocasazza:pid-namespace-stale-lock
Closed

ocasazza wants to merge 2 commits into
project-minigraf:mainfrom
ocasazza:pid-namespace-stale-lock

Conversation

@ocasazza
Copy link
Copy Markdown
Contributor

Closes #317.

Problem

Under Linux PID namespaces (Kubernetes), every container's main process is PID 1. When a pod dies and a replacement pod mounts the same persistent volume, the stale .graph.lock names the new process's own PID, so:

  1. pid == our_pid → acquire refuses with "Database is already open in this process" (the Transient 'Page N out of bounds (total pages: N-1)' write failure at large file sizes #304 behavior), and
  2. is_process_alive(1) is always true, so the lock never self-heals.

The database is bricked after every pod restart until someone deletes the lock file by hand. This is a confirmed production incident for us.

Fix

The lock now records a per-process-incarnation token alongside the PID. On procfs platforms (Linux/Android) the lock content is <pid>:<starttime>, where starttime is field 22 of /proc/<pid>/stat (clock ticks since boot) at acquire time. Two processes in different PID namespaces may share a PID number, but not a start time.

In acquire:

  • pid == our_pid branch: if the recorded start time is present, differs from our own, and procfs is available, the holder is a dead namespaced predecessor → remove + retry, same as the dead-foreign-PID path. If the start time matches, is absent, or is unreadable, the Transient 'Page N out of bounds (total pages: N-1)' write failure at large file sizes #304 refusal is kept exactly as-is.
  • Foreign-PID branch: a live foreign PID can also be a namespace alias for a dead holder (the previous container's process happened to share an in-namespace PID with one of ours). When both start times are known and differ, the live process is not the lock's author and the lock is reclaimed.
  • Non-procfs platforms (macOS/Windows): process_start_time returns None, so both branches fall through to today's conservative behavior unchanged.

Backward compatibility

  • Lock files containing a bare <pid> (written by every released minigraf version) still parse via parse_lock_content; with no start time recorded, both branches behave exactly as before.
  • Nothing else in the crate parses lock content (it is diagnostics-only), and the lock file is still deleted on drop as before.

Invariants preserved (#304)

test_second_open_in_same_process_is_refused and test_reopen_after_drop_still_works are unchanged and pass: a lock naming our PID with a matching (or absent, or unreadable) start time is still refused, and a refused acquire still leaves the existing lock in place.

Tests

Added in src/storage/backend/file.rs, mirroring the existing FileLock test style:

  • test_namespaced_same_pid_lock_with_wrong_start_time_is_reclaimed (procfs-gated) — our PID + wrong start time must be reclaimed
  • test_same_pid_lock_with_our_own_start_time_is_refused (procfs-gated) — our PID + our own start time must still be refused
  • test_legacy_bare_pid_lock_with_our_pid_is_refused — backward compat for pre-start-time lock files

Verification

  • cargo test — full workspace suite passes (macOS)
  • cargo test --lib storage::backend::file — 11/11 pass on Linux (podman, rust:1-slim), including the two new procfs-gated tests and the pre-existing test_dead_other_process_lock_is_still_reclaimed
  • cargo clippy -- -D warnings — clean
  • rustfmt --check on the changed file — clean

CHANGELOG entry added under Unreleased.

Under Linux PID namespaces every container's main process is PID 1, so
when a pod dies and a replacement pod mounts the same persistent volume,
the stale lock file names the new process's own PID. acquire() refused
it as "already open in this process" (project-minigraf#304 semantics), and
is_process_alive(1) is always true, so the database stayed locked until
someone deleted the lock file by hand.

On procfs platforms the lock now records the holder's process start time
(/proc/<pid>/stat field 22) as <pid>:<starttime>. A lock naming our own
PID whose recorded start time differs from ours is a dead namespaced
predecessor and is reclaimed; the same disambiguation applies to a live
foreign PID that aliases a dead holder. A matching, absent, or
unreadable start time keeps the project-minigraf#304 refusal exactly as-is, and legacy
bare-<pid> lock files and non-procfs platforms are unchanged.
@ocasazza
ocasazza requested a review from adityamukho as a code owner August 14, 2026 22:27
@codecov
codecov Bot commented Aug 15, 2026
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.18519% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/storage/backend/file.rs 85.18% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@adityamukho
Copy link
Copy Markdown
Collaborator

Thank you for the patch. Your analysis is correct and the defect is real. I accept issue #317. But I will not merge this mechanism. The reasons follow.

Correct parts of the patch

  • The code reads field 22 of /proc/<pid>/stat correctly. It divides the line at the last ) character. A comm value that contains a space or a parenthesis therefore causes no error. I tested this against real /proc data.
  • The code also parses an old lock file that contains only a PID. This keeps compatibility.
  • On a platform without procfs, the function returns None. macOS and Windows therefore keep the present behavior.
  • The foreign-PID branch is an improvement inside one PID namespace. When a process dies, a different process can receive the same PID number. The old code could not detect this condition. A comparison of the start times detects it correctly.

Why I cannot merge the patch

The patch assumes that two conditions together identify a dead process:

  1. The PID in the lock file is equal to our PID.
  2. The start time in the lock file is different from our start time.

This assumption is not correct. In a different PID namespace, /proc/<pid> does not show the holder of the lock. It shows a different process in your own namespace, or it shows your own process. A different start time shows only that the holder is not the process that you can see. It does not show that the holder is dead. The start time indicates liveness correctly only inside one PID namespace. This defect occurs only between different PID namespaces.

Example

Two containers in one pod use the same volume. The pod does not set shareProcessNamespace.

  1. Container A opens the database. Its PID is 1. It writes 1:<S_A> to the lock file.
  2. Container B opens the same database. Its PID is also 1. It reads 1:<S_A>. Its own start time is S_B. The two start times are different.
  3. Container B decides that the holder is dead. It deletes the lock file of container A. Container A is still alive.

Two FileBackend objects now use one file. Each object keeps its own value of header.page_count. This is the same divergence as #304, but between two processes. The check for one process cannot detect it. The first object that closes deletes the lock file of the other object.

The same sequence occurs during a rolling update on an RWX volume (NFS, EFS, CephFS or Azure Files). The old pod continues to operate while the new pod starts. Most rolling updates operate in this manner.

The present code refuses the lock in this condition. The message is not clear, but no data becomes corrupt. With this patch, the data becomes corrupt and the user receives no message. The lock prevents corruption between processes. A refusal is therefore the correct failure mode. I prefer to keep the availability defect. See PHILOSOPHY.md, section 6.

Other items

  • The acquire function now calls itself from two locations. If remove_file continues to fail, the function calls itself until the stack is full. The user receives no error message. Two processes in different PID namespaces can also each decide that the other process is dead. The two processes then delete the lock file of each other continuously. A loop with a maximum number of retries prevents both conditions. One of these two locations is in the old code. You did not introduce the defect, but the patch makes it more probable.
  • The code at lines 76 to 81 prints holder.trim() after the text holder PID:. The message therefore shows holder PID: 1:9.
  • The three new tests examine only the conditions that must succeed. No test examines a live foreign process that has the same start time. That condition must cause a refusal.

My recommendation

Use a lock from the operating system, as SQLite does. Use flock with LOCK_EX and LOCK_NB, or fcntl with F_OFD_SETLK. Hold the lock on the .graph file while the backend is open. The kernel releases the lock when the process dies. A stale lock therefore cannot occur. The code compares no PIDs and reads no data from /proc.

This method corrects #317, #304 and the PID-reuse condition. It needs less code than the present lock file.

Two additions are necessary. The kernel gives these locks to a process, not to a handle. A second flock in the same process therefore succeeds. A global set of canonical paths detects the #304 condition directly. The .lock file then becomes a diagnostic file only, and the format of its contents is no longer important.

libc is already a dependency through uuid and getrandom. The size of the binary therefore does not increase.

Two conditions need attention. On NFS, Linux uses fcntl to emulate flock from version 2.6.37. This operates correctly on NFSv4, but NFSv3 needs lockd. The code must detect ENOTSUP and then use the present behavior. Windows needs the equivalent function, LockFileEx.

Please tell me if you want to write this patch. You have a reproduction in production and I do not. If you do not want to write it, I will do it. Thank you again for your analysis in #317. It made the verification easy.

@ocasazza
ocasazza commented Aug 21, 2026 •
Copy link
Copy Markdown
Contributor Author

Hi @adityamukho, your explanation is well over my head; I develop more in user space. But I appreciate the opportunity to learn; thank you for the detail here

Because I don't have the experience to carry the patch through / and I am unfamiliar with the preferred change management policies of this project; please proceed with patching however you see fit - I will observe. Maybe I can be of more assistance in the future

@adityamukho
Copy link
Copy Markdown
Collaborator

@ocasazza Thank you for being candid about this. I will close this PR now, and take #317 forward in a fresh PR because it involves redoing almost everything from scratch, except perhaps some of the tests.

I will be sure to credit you for the work I pick from here, as a co-author.

Your report made the verification easy and you supplied the reproduction, the mechanism and the reference to #304. That already covers the most difficult part of the work.

Thank you!

adityamukho added a commit that referenced this pull request Aug 22, 2026
* docs: add kernel file locking design spec (#317, #304)

Replaces the PID-sidecar FileLock with std::fs::File::try_lock. A PID is
not a liveness token: it is not unique across PID namespaces and is
recycled within one, so any PID-based liveness check is unsound wherever
containers run.

Supersedes PR #318, whose /proc start-time comparison cannot establish
liveness across namespaces and would convert a safe refusal into silent
cross-process corruption.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* docs: add kernel file locking implementation plan

Scoped to the core fix plus the PID-namespace regression test. The nightly
NFS and Docker tiers are tracked separately in #324.

Also corrects two errors in the spec: OpenOptions exposes both public
fields and builder methods, so allow_unlocked needs both; and there are 14
crash-simulation call sites, not 23 -- the earlier figure counted comment
mentions of mem::forget alongside the actual calls.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* chore: ignore .superpowers/ scratch workspace

Subagent-driven-development writes its ledger, briefs, and review packages
there. It is scratch, not source.

* docs: fix four plan defects found in the pre-flight scan

- Task 4's crash assertion required death by signal, which fails on
  Windows CI where abort() yields an exit code instead. wal_test.rs is
  gated only on not(wasm32), so it does run there.
- Task 5's Files block omitted examples/pid_ns_helper.rs, which its own
  Step 2 creates.
- Task 5's catching-power check used `git stash`, which cannot revert work
  already committed by Tasks 1-2 -- as written it would have tested the
  fixed code against itself and always passed. Uses a throwaway worktree
  at the pre-fix commit instead.
- Task 6 edited .wiki/, which does not exist in a worktree; it is a plain
  clone living only in the main checkout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* feat(storage): add lock-decision classifier and declare MSRV 1.89

Isolates the try_lock decision from the I/O so the unsupported-filesystem
branch is testable without a filesystem that cannot lock.

allow_unlocked deliberately does not apply to WouldBlock.

* fix(storage): lock the .graph file in the kernel, not a PID sidecar

A PID is not a liveness token. It is not unique across PID namespaces and
is recycled within one, so the sidecar could never distinguish a crashed
holder from a live one. A container killed with SIGKILL left the sidecar
holding pid 1, and the replacement container -- also pid 1 -- refused to
open the database forever (#317).

File::try_lock puts the decision in the kernel, which releases the lock on
process death however it occurs. Because flock and OFD locks attach to the
open file description rather than the process, this also denies a second
open within one process, covering #304 with no bookkeeping.

A leftover sidecar from v1.2.x is ignored, never deleted: a still-running
old process may depend on it.

* fix(storage): field drop order, ProceedUnlocked registry, message repair

Three fixes to the kernel-file-lock work for #317/#304:

1. `_path_guard` is now declared before `file` in `FileBackend`. Rust drops
   struct fields in declaration order, so `file` (and the kernel lock it
   holds) was previously released before the registry entry was removed,
   opening a window for another thread to acquire the lock and insert its
   own entry that this guard's drop would then incorrectly delete.

2. `LockOutcome::ProceedUnlocked` now consults `already_open_here` and
   inserts into `OPEN_PATHS` before returning, matching the `Acquired` arm.
   On an unlockable filesystem there is no kernel lock, so the registry is
   the only thing preventing a second same-process handle to the same path
   from corrupting the file with two independent page tables (#304).

3. Repaired the `ProceedUnlocked` bail! message: its line-continuation
   backslashes had been dropped, leaving literal runs of spaces at each
   line join. Restored the `\`-continuation style used by the neighbouring
   bail! calls so it reads as one clean sentence, textually identical to
   the `LockOutcome::Held` same-process message.

`test_refused_open_does_not_corrupt_path_registry` was rewritten to assert
on the second refusal's error message rather than just its Err-ness.

Not done: an in-process test for "a refused open leaves drop-removal
intact" is not attempted here. That property is only observable with an
independent process holding the kernel lock (Task 5's subprocess
machinery) — inside one process, once the holder drops, try_lock simply
succeeds and the registry is never consulted.

Also not added: a test that reaches `ProceedUnlocked`'s same-process
refusal directly through `FileBackend::open_with`. That requires
`File::try_lock()` to return a real `TryLockError::Error` from an
actually unlockable filesystem (e.g. NFSv3 without lockd) — not
reproducible with a tempdir on a normal CI runner. The `classify` unit
tests already pin that `ProceedUnlocked` is produced correctly from
that raw `Result`; the registry-consulting behavior inside the
`open_with` match arm has no unlockable-filesystem-free way to exercise.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* feat(db): add OpenOptions::allow_unlocked

Fails closed by default when a filesystem cannot lock. The escape hatch is
for single-writer-by-convention deployments on mounts without working
locks; it never bypasses a lock another handle holds.

* docs: verify clippy the way CI does, not more strictly

Task 7 called for --all-targets, which lints test binaries carrying
several hundred pre-existing failures on main. CI runs --all-features.
Gating the branch on debt CI ignores would block it for unrelated
reasons.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* fix: gate FileBackend::open on tests and rewrite stale comment

- Gate FileBackend::open with #[cfg(test)] since production code now calls
  open_with directly. Add comment explaining this survives for test convenience.
  Fixes clippy dead-code warning that would fail CI.
- Rewrite checkpoint comment to remove stale 'manual lock deletion' example
  from .graph.lock sidecar era. Now accurately describes legitimate use cases:
  filesystems that cannot lock at all (NFSv3 without lockd, FUSE mounts).

* chore: ignore memory.graph FTS sidecar files

.gitignore covered memory.graph but not its FTS index sidecars, so a
broad `git add` swept 72KB of local sqlite binaries into a commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* test: simulate crashes with real subprocesses

mem::forget no longer models a crash: it leaks the File, so the kernel
lock stays held for the life of the test process and no external cleanup
can release it.

A child that aborts is both the accurate model -- the kernel closes its
descriptors exactly as it would for a SIGKILLed container -- and the one
that needs no test-only hook in the shipping crate. #[cfg(test)] cannot
reach tests/, and #[doc(hidden)] pub would be callable in production.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* fix(storage): retry WouldBlock with bounded backoff before failing open

Command::spawn forks the whole process before exec'ing the child; until
the child reaches execve and its close-on-exec descriptors close, it
transiently holds a duplicate of every flock its parent has open. A
concurrent try_lock() elsewhere -- another thread in this process, a
sibling process, or bindings that shell out (Python/Node commonly do)
-- can observe WouldBlock during that window even though no other
process is really contending for the file. This surfaced as flaky
"Database is locked by another process" failures on completely
unrelated tests once tests/wal_test.rs started spawning real crash
subprocesses.

SQLite solves the same class of problem with busy_timeout; this is our
version: retry try_lock() up to 10 times with backoff (5ms doubling,
capped at 50ms, ~375ms worst case) before concluding another process
genuinely holds the lock. The retry is skipped entirely when this
process already has the path open -- that conflict is unwinnable by
waiting, since we are the only thing that could release it -- so
test_second_open_in_same_process_is_refused stays instant.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* test: restore explicit-transaction crash coverage, add cross-process lock test

Review round 2 fixes:

1. test_explicit_tx_all_or_nothing_commit and
   wal_recover_committed_tx_crash_before_checkpoint had been converted to
   plain db.execute() calls in the crashing child, silently dropping the
   one thing they existed to pin: that WriteTransaction::commit() itself
   writes to the WAL before a crash, and that a multi-statement commit
   recovers atomically (all facts, not a subset). Neither property was
   asserted anywhere else in the tree. Added a CrashTx::{Implicit,Explicit}
   mode to run_crashing_child/crash_child_entrypoint (env var
   MINIGRAF_CRASH_TX) so the child can wrap statements in begin_write()/
   commit() instead of auto-committing each one, and restored both sites
   to CrashTx::Explicit.

2. The retry-on-WouldBlock fix in FileBackend::open_with had no test:
   nothing on the branch spawned a second process to actually hold the
   lock while another tried to open it, so the cross-process error
   message and the retry-budget bound were both unexercised -- and an
   already_open_here that wrongly returned true would have passed the
   whole suite. Added a lock-holding child (hold_lock_entrypoint) and a
   driver test that asserts: the open fails, it fails within one second
   (not by hanging out the full retry budget plus the holder's longer
   sleep), the error is the cross-process wording and not the
   same-process one, and a subsequent open succeeds once the holder
   exits.

Also: corrected the OPEN_PATHS doc comment (it now also gates the retry
decision, not just error wording) and the LOCK_RETRY_ATTEMPTS comment
that claimed a genuinely locked database "fails fast" (it pays the full
~375ms budget). Noted CRASH_CHILD_SPAWN_LOCK is now belt-and-braces now
that the real fix lives in open_with.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* test: reproduce #317 across PID namespaces

Two processes at PID 1 in separate namespaces sharing one file, via
unshare -- no container runtime needed. Verified to fail against the
pre-fix code: the killed holder's sidecar left pid 1 behind and the
replacement was refused permanently.

The second test guards the other direction: two live holders must still
be refused, which is what PR #318's start-time comparison would have
broken.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>

* test: fail loudly on a missing helper, kill-on-drop for container A

Review round 1 fix for #317's regression test.

A missing helper binary used to be a silent skip. Under any filtered
`cargo test` invocation that never triggers a rebuild, that reports a
green, vacuous pass with nothing in the output to say so. Now the skip
is reserved for the case that actually warrants it -- unshare_prefix()
returning None, i.e. this environment cannot create PID namespaces at
all. If namespaces work but the helper binary is absent, that can only
mean `cargo build --examples` was not run, so the test panics with a
message naming the binary instead of asserting nothing.

Also wrap container A's Child in a kill-on-drop guard in both tests.
std::process::Child does not kill its child on Drop, and both tests
hold container A across an assert! before their explicit kill/wait; if
that assert (or the wait_with_output().expect() further down in the
second test) panics first, the panic used to unwind straight past
cleanup and orphan container A -- PID 1 in its own namespace, still
mid-sleep, still holding the flock. The guard makes that unwind-safe
without restructuring either test.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>

* docs: record the kernel locking change

CHANGELOG carries the semver-relevant facts (new public option, MSRV
1.85 -> 1.89, behaviour change on unlockable filesystems, Windows
mandatory locks, bounded cross-process retry) so the release decision
can be made later.

Adds STG-025 through STG-027 with the actual error text; the lock
errors had no reference entries. Updates docs/TEST_COVERAGE.md,
CLAUDE.md, and README.md to the measured count (1021 tests: 1013
passing, 8 ignored) and reconciles the itemized breakdown -- including
the Wave 3 Conclusion section and the XTDB/Datomic compatibility
section headers, command-reference counts, and bullet lists (now one
accurate bullet per real test, matched in both directions against the
source), which had all drifted stale before this branch -- so no count
in the file contradicts another. Documents that hold_lock_entrypoint's
child process leaks a stray harness summary line into cargo test
output (easy to mistake for a second partial run of the lib unit test
binary), and gives a verified filter to exclude it when totalling.
Fixes a stale "Diagnostic only." comment on already_open_here in
src/storage/backend/file.rs -- its return value now also gates the
WouldBlock retry loop.

* style: apply cargo fmt to test files

cargo fmt --check failed on three test files touched by the kernel
file locking work (migration_matrix_test.rs, pid_namespace_test.rs,
wal_test.rs) — formatting only, no behavior change.

Task 7 verification (docs/superpowers/plans/2026-08-22-kernel-file-locking.md).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* fix(storage): make the unlocked-path registry check atomic

Final-review fix wave, four items.

1. TOCTOU in the ProceedUnlocked arm. already_open_here() took OPEN_PATHS,
   read, and released it; the insert then took the lock again. Two threads
   opening the same path on a filesystem that cannot lock both observed
   "absent", both inserted, and both proceeded -- two FileBackends on one
   file with diverging page_count, which is exactly the #304 corruption.
   That arm has no kernel lock behind it, so the registry was the only
   guard, and its own comment said so while not providing it. Now one
   critical section using HashSet::insert's return value as test-and-set.
   A poisoned mutex fails closed.

2. Record the OpenOptions semver break in the CHANGELOG. Adding a public
   field to a struct that is not #[non_exhaustive] breaks struct-literal
   construction downstream. The release decision is deferred, so this
   documents the facts needed to make it: first post-1.0 field addition,
   the v0.19.0 precedent predates the guarantee, and #[non_exhaustive]
   would immunise future additions at the cost of being breaking itself.

3. Remove a flake window. The cross-process lock test held for 600ms
   against a ~375ms retry budget -- ~215ms of slack, across three CI
   platforms with 34 test binaries in flight. Now 3s, and the holder is
   killed rather than waited out, which is both faster than the original
   and tests release-on-SIGKILL instead of release-on-clean-exit.

4. Mark tiers 2 and 3 in the design spec as deferred to #324, and state
   plainly that this leaves the spec's own NFS silent-no-op risk
   unmitigated on merge.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* docs: correct two factual errors found in the fix-wave re-review

- The CHANGELOG said this branch updated "three" exhaustive OpenOptions
  literals. It updated two: the Default impl and one test. Only the test
  is a "downstream literal" in the sense the semver paragraph means.
  Verified by walking every OpenOptions literal in the tree.

- Two statements in the design spec still read as though Tier 2 ships
  with this branch: the prose at the NFS discussion, and the risk-register
  row, which a skim-reader may see without ever reaching the delivery
  note. Both now say plainly that the mitigation is deferred to #324 and
  the risk is unmitigated on merge -- and the risk row now notes why
  fail-closed does not cover it: a silent no-op returns Ok(()), so there
  is no error to fail on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* docs: record kernel file locking in ROADMAP and CLAUDE.md status

Both were missed in the Task 6 doc sync. ROADMAP's release history
described v1.2.2's #304 fix with no note that this branch supersedes the
mechanism behind it, and CLAUDE.md's status line — which loads every
session — did not mention the change or the MSRV bump.

Neither states a version: the release decision is deferred, and both
point at the CHANGELOG's breaking-changes section instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* test: kill the namespaced holder by process group, not by pid

The #317 regression test passed locally and failed on GitHub's ubuntu
runner. Both tests ran (no skip), and the two-live-holders test passed,
so namespaces work there — the difference is how container A gets killed.

Command::kill() signals only the process we spawned. When `unshare` is
that process, `--kill-child` covers the rest: it sets PR_SET_PDEATHSIG on
the process that becomes PID 1. But GitHub's runners restrict unprivileged
user namespaces, so the probe selects the `sudo` fallback, and then a
privilege wrapper sits in between. SIGKILL runs no handler, so the wrapper
forwards nothing, `unshare` and its namespaced child survive, and the lock
is still held when container B tries to open.

That failure is maximally misleading: the #317 test fails identically
whether or not the bug is present, because the lock genuinely is still
held. A regression test that cannot distinguish those two states is worse
than no test.

The holder now leads its own process group and is killed as a group, with
the same privilege escalation the prefix used — root-owned descendants
reject an unprivileged signal with EPERM. The second test uses the same
path: its kill is cleanup only and cannot fail the assertion, but an
orphan there leaves a namespaced process on the flock for 300 seconds,
which on a shared runner breaks somebody else's test.

Both asserts now print the prefix actually selected and container B's
stdout and stderr, so the next failure of this kind names its own cause
instead of requiring a bisect against CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* docs: credit @ocasazza in the CHANGELOG for #317

Attribution existed in the commit trailers, the design spec, and the PR
body, but not in the CHANGELOG — which is the one that ships to users and
gets read at release time.

They supplied the reproduction, identified the PID-namespace mechanism,
and traced the refusal back to the #304 fix that caused it. That analysis
is why this was quick to verify rather than quick to guess at.

The co-author trailer is repeated here deliberately. This repository
squashes with squash_merge_commit_message=COMMIT_MESSAGES, so the squash
body is every commit message concatenated in order, and git only parses
trailers in the FINAL paragraph. The trailers on a1bfec1 and 8cc1419 land
mid-body and would read as prose. Carrying it on the last commit puts it
where it will actually be parsed.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* test: let the namespaced holder end its own life

CI named the cause once the assertions printed it:

  prefix used: ["sudo", "-n", "unshare", "--pid", "--fork", "--mount-proc",
                "--kill-child"]
  container B stdout: OPEN_ERR Database is locked by another process

GitHub's runners restrict unprivileged user namespaces, so the probe takes
the sudo fallback. Ubuntu's sudo defaults to use_pty, which runs the
command in a NEW SESSION. Signalling the process we spawned reaches only
sudo, and SIGKILL runs no handler so nothing is forwarded; killing the
process group misses the holder too, because it is no longer in that
group. The previous attempt at a group kill failed for exactly this
reason.

Killing across that privilege boundary cannot be made reliable from here,
so stop trying. The holder now aborts itself: no unwinding, no Drop, no
checkpoint, no library-side lock release -- the same state a SIGKILLed
container leaves, which is what #317 is about. Test 1 asks for a zero
hold, so A dies the moment it holds the lock; test 2 asks for a bounded
2s hold, long enough for B to be refused while A lives and short enough
that a failure cannot strand a namespaced process on the flock.

KillOnDrop goes with it. It existed to stop a panic orphaning a holder
mid-way through a 300-second sleep; holders are now self-limiting, so it
guards nothing.

Catching power re-verified after the rework, not assumed: against pre-fix
e056905 the regression test still fails with the #317 message while the
two-live-holders test still passes -- the same discrimination as before.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* test: let the parent decide when the live holder releases

CI moved the failure from test 1 to test 2: the #317 regression test now
passes on the runner, and the two-live-holders test failed with container
B reporting OPEN_OK where it must be refused.

Cause was the fixed 2s hold. Container B has to start through
`sudo unshare` and attempt an open, and how long that takes differs
enormously between a laptop and a loaded runner. A hold that is generous
locally expires mid-attempt on CI, B then legitimately succeeds because
nothing is holding the lock any more, and the test reports that a live
holder was not excluded — which was never what happened.

Any fixed number is a guess at someone else's machine. The holder now
waits for a release marker the test writes once B's result is recorded,
so the hold is exactly as long as it needs to be. The helper keeps a 60s
deadline purely as a backstop, so a panic in the parent cannot strand a
namespaced process on the flock.

Faster too: 0.64s against 2.16s, because nothing waits out a sleep that
was mostly slack.

Catching power re-verified against pre-fix e056905 with a freshly built
helper -- the regression test fails there with the #317 message while the
two-live-holders test passes.

Note for anyone iterating on this: `cargo build --tests` and
`cargo test --test <name>` do NOT rebuild `examples/`, so the helper can
go stale and silently change behaviour under you. Plain `cargo test`,
which is what CI runs, does rebuild it.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* docs: resync test counts after main was merged in

The merge brought #325 and #326, adding two lib unit tests, so every count
this branch had just synced was stale again: 1015 passing, 8 ignored,
1023 total, with 678 in the lib binary.

Updated CLAUDE.md, README.md, and docs/TEST_COVERAGE.md, including the
itemized Test Summary — its 33 bullets sum to exactly 1023, verified
rather than eyeballed.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* test: pass the release marker as an argument, not an env var

sudo defaults to env_reset: everything outside a small allowlist is
stripped before the command runs. GitHub's runners restrict unprivileged
user namespaces, so the test takes the `sudo unshare ...` fallback, and
the holder never saw MINIGRAF_NS_RELEASE_MARKER at all. It took the
"nothing to wait for" path and aborted the instant it held the lock, so
container B found the file free and opened it — and the test reported
that a live holder had not been excluded, when there had been no live
holder to exclude.

This also corrects the previous commit's diagnosis. That one blamed a
fixed 2s hold expiring on a slow runner. The hold was never the problem:
MINIGRAF_NS_HOLD_MILLIS was stripped by the same mechanism, so A aborted
immediately in that version too. The release marker was the right change
for a different reason — it removes the guess about someone else's
machine — but the failure it was meant to fix had another cause entirely.

Arguments cross the sudo boundary; environment variables do not.

Verified locally under `env -i`, which reproduces env_reset exactly: the
holder stays alive with a completely stripped environment and aborts only
once the marker appears. That is the CI condition, reproduced on a machine
with no passwordless sudo.

Catching power re-checked against pre-fix e056905: the regression test
still fails there with the #317 message, the two-live-holders test still
passes.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

* fix(test): read the raw header after closing the database

Windows CI failed with os error 33, "another process has locked a portion
of the file", on a read whose "other process" was the test itself.

Windows file locks are mandatory, not advisory, and they exclude every
handle but the one holding them — a second handle in the same process
included. test_manual_checkpoint_deletes_wal opened the .graph file
through a fresh `File` to assert the header reflected the checkpoint,
while the database was still open and holding the lock. On Unix the lock
is advisory and that read succeeds, which is why this only ever failed on
one leg of the matrix.

Moving the read after the existing `drop(db)` fixes it and asserts the
same thing. Two other raw reads in the suite already closed their handle
first, so this was the only site.

Also corrects the CHANGELOG, which claimed only that "other processes"
cannot read an open database on Windows. That understated it: the
exclusion covers your own process's other handles, which is the case a
user is far more likely to hit — inspecting a file they themselves have
open. The wiki carried the same understatement and is corrected in its
own repository.

Co-Authored-By: Olive Casazza <olive.casazza@schrodinger.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dtz3m9Ww7SUbft1TqRSGXf

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Olive Casazza <olive.casazza@schrodinger.com>
@adityamukho
Copy link
Copy Markdown
Collaborator

A correction to something I told you in this thread.

I wrote: "The kernel gives these locks to a process, not to a handle. A second flock in the same process therefore succeeds." That statement is wrong.

flock attaches a lock to the open file description, not to the process. fcntl with F_OFD_SETLK does the same. If one process calls open() twice on one file, it gets two different open file descriptions, and the second lock request is refused. I tested this before I wrote the patch, and the kernel refused the second request in both cases. Only the older fcntl record locks (F_SETLK) behave in the manner I described.

This changes one part of my recommendation. I told you that we would need a process-global set of paths to detect the #304 condition, because the kernel would not. That is not correct: the kernel detects it. The code keeps a small set of paths, but only to select between two error messages. If it were removed, the database would still refuse the second handle.

The correction does not change the conclusion about your patch. A start time still cannot show liveness across PID namespaces.

The fix is now in main. Three items from this thread are in it:

  1. Your reproduction is the regression test. It uses unshare to put two processes at PID 1 in different namespaces. I confirmed that the test fails against the code before the fix, and passes after it.
  2. Your test for a stale same-PID lock is the ancestor of that test. The commit gives you credit.
  3. The CHANGELOG names you as the person who reported and diagnosed the defect.

Two things that CI found, in case they are useful to you. Both are relevant to anyone who writes tests with subprocesses.

  • GitHub runners restrict unprivileged user namespaces, so my test used sudo unshare. sudo removes almost all environment variables before it runs a command. My test passed a parameter in an environment variable, and the child never received it. The child then did the wrong thing silently, and the test reported a fault that had not occurred. Command-line arguments cross that boundary; environment variables do not.
  • Windows file locks are mandatory, not advisory, and they exclude every other handle — including a second handle in the same process. One test read the database file directly while the database was open. On Windows this failed with the message "another process has locked a portion of the file". The other process was the test itself.

Thank you for the report. The analysis in #317 is what made this quick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FileLock refuses stale lock under PID namespaces (Kubernetes): every container is PID 1

2 participants

0