8000 Block store writes racing user deletion by jasonmadigan · Pull Request #161 · tracefinity/tracefinity · GitHub
[go: up one dir, main page]

Skip to content

Block store writes racing user deletion - #161

Merged
jasonmadigan merged 6 commits into
mainfrom
160-block-store-writes-after-user-deletion
Jul 30, 2026
Merged

jasonmadigan merged 6 commits into
mainfrom
160-block-store-writes-after-user-deletion

Conversation

@jasonmadigan
@jasonmadigan jasonmadigan commented Jul 30, 2026 •
Copy link
Copy Markdown
Collaborator

Summary

  • Deletion now closes the evicted store instances (session/tool/bin/project); a closed store refuses writes and drops its in-memory data, so a store reference captured by an in-flight threadpool request (e.g. generate_stl) can no longer rewrite or serve the deleted user's data.
  • get_stores/get_project_store and the deletion sequence (evict, close, rmtree) serialise on a per-user threading.Lock, closing the residual Evict _project_store_cache on user deletion #159 window where a store could be built from files mid-rmtree and cached with stale data.
  • Non-store file writes are guarded too: the tracer mask write, upload, trace-mask and shared STL output paths check the captured store before writing, so an in-flight request cannot recreate the deleted user's tree.
  • Closed-store writes raise a typed StoreClosedError, mapped to a 410 at the API layer, so requests losing the race fail cleanly instead of as unhandled 500s.
  • Regression tests: captured-store write after deletion for all four stores (asserts StoreClosedError and nothing reappears on disk, including after dir recreation), store creation racing an event-gated slow rmtree, and a delete landing mid-trace (asserts the user dir and mask never reappear).

Approach

The issue offered a per-user lock or a deleted-generation check in _save. Neither alone closes the race, so this uses both halves, each for the window only it can cover:

  • A closed flag checked in the store write path handles captured references. A lock alone cannot: the offending write happens after deletion has completed and released any lock. The check is atomic with the write because set()/delete() hold the store's existing lock for check + mutate + save, and close() takes the same lock, giving a clean happens-before (write lands before rmtree, or the write sees closed and refuses). close() also drops the in-memory dicts, belt and braces with the flag, so a captured reference cannot read stale records either; in-flight reads racing deletion were already best-effort, so nothing meaningful is lost.
  • The per-user lock handles cache repopulation during deletion. A closed flag alone cannot: a fresh instance created between eviction and rmtree loads the old files and is not closed.

Non-store file writes get the same treatment, with the captured (now closed) store as the deletion signal. The seconds-long window is a trace awaiting its model call: the tracer used to mkdir + write the mask when the call returned, silently recreating the deleted user's dir before the store set() raised, orphaning the mask with no cleanup. trace_tools now passes the captured session store's ensure_open into the tracer as a before_mask_write hook that aborts the write, and the upload, trace-mask and shared STL generation paths check the captured store before writing into the user tree. Fresh requests for a deleted user id still recreate dirs by design.

Refused writes raise StoreClosedError (a RuntimeError subclass shared in app/services/store_errors.py), mapped to a 410 with a terse detail by an app-level exception handler. The delete handler itself is now a sync def: it takes threading locks and runs rmtree, which belongs in FastAPI's threadpool, not on the event loop.

Locks are never removed from _user_locks (popping reintroduces races); the dict is bounded by user ids seen, fine for the single-process scope here. Multi-process remains #155. drawer_store.py is untouched: nothing instantiates it.

Test evidence

  • pytest (backend, main checkout venv): 298 passed
  • make lint-backend: passes (frontend untouched; worktree lacks node_modules for eslint)
  • Store tests fail on the base branch: silent resurrection of sessions.json/tools.json/bins.json/bin-projects.json reproduced before the fix
  • The mid-trace test fails with the before_mask_write hook disabled: the orphaned mask and recreated user dir reproduce

Closes #160

A store reference captured by an in-flight request survives user
deletion and its next _save() rewrites the whole dict to disk,
resurrecting deleted data. close() marks the instance dead under the
store lock; _save() then refuses to write. Fresh instances created
after deletion start empty from the wiped directory.

Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
Close the two remaining single-process windows around DELETE /users/me:

- captured references: deletion now closes the evicted store instances,
  so a threadpool request that grabbed its stores before deletion gets
  RuntimeError instead of silently rewriting sessions/tools/bins/
  projects to disk after ensure_user_dirs recreates the directory
- cache repopulation: get_stores/get_project_store and the deletion
  sequence (evict, close, rmtree) now serialise on a per-user lock, so
  a store can no longer be built from files mid-rmtree and cached with
  stale data

Regression tests cover a captured-store write after deletion for all
four stores and store creation racing a slowed rmtree.

Closes #160

Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
@jasonmadigan
jasonmadigan force-pushed the 160-block-store-writes-after-user-deletion branch from 6f033b3 to b7ad063 Compare July 30, 2026 19:58
delete_user_data acquired threading locks and ran rmtree in an async
handler, freezing the event loop for every request whenever deletes
contended. A plain def handler runs in FastAPI's threadpool where
blocking is safe. Lock ordering (user lock, then store lock) is
unchanged and consistent across threadpool workers.

Per review feedback on #161.

Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
Replace the bare RuntimeError raised by closed stores with
StoreClosedError (shared in app/services/store_errors.py) and register
an exception handler so in-flight requests losing the race against
user deletion get a 410 instead of an unhandled 500. set()/delete()
now check the flag before mutating so a refused write cannot leave a
phantom record in memory.

Per review feedback on #161.

Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
A closed store kept serving the deleted user's records from memory to
any captured reference. Dropping the dicts in close() (belt and braces
with the closed flag) makes reads return empty rather than stale data;
in-flight reads racing deletion were already best-effort, so nothing
meaningful is lost.

Per review feedback on #161.

Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
The store write guard did not cover plain file writes. A trace awaiting
its model call could outlive a DELETE /users/me: the tracer then ran
mkdir(parents=True) and wrote the mask into a recreated dir, after which
the store set() raised and the file was orphaned. The captured (now
closed) store is the signal: trace_tools passes ensure_open into the
tracer as a before_mask_write hook, and the upload, trace-mask and
shared STL generation paths check it before writing into the user tree.
Fresh requests for a deleted user id still recreate dirs by design.

Regression test simulates a delete landing mid-trace and asserts the
user dir and mask never reappear.

Per review feedback on #161.

Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
@jasonmadigan
jasonmadigan marked this pull request as ready for review July 30, 2026 20:40
@jasonmadigan
jasonmadigan merged commit a9c5306 into main Jul 30, 2026
4 checks passed
@jasonmadigan
jasonmadigan deleted the 160-block-store-writes-after-user-deletion branch July 30, 2026 20:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

In-flight store references survive user deletion

1 participant

0