Repository navigation
Block store writes racing user deletion - #161
Merged
Merged
Conversation
A store reference captured by an in-flight request survives user deletion and its next _save() rewrites the whole dict to disk, resurrecting deleted data. close() marks the instance dead under the store lock; _save() then refuses to write. Fresh instances created after deletion start empty from the wiped directory. Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
Close the two remaining single-process windows around DELETE /users/me: - captured references: deletion now closes the evicted store instances, so a threadpool request that grabbed its stores before deletion gets RuntimeError instead of silently rewriting sessions/tools/bins/ projects to disk after ensure_user_dirs recreates the directory - cache repopulation: get_stores/get_project_store and the deletion sequence (evict, close, rmtree) now serialise on a per-user lock, so a store can no longer be built from files mid-rmtree and cached with stale data Regression tests cover a captured-store write after deletion for all four stores and store creation racing a slowed rmtree. Closes #160 Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
jasonmadigan
force-pushed
the
160-block-store-writes-after-user-deletion
branch
from
July 30, 2026 19:58
6f033b3 to
b7ad063
Compare
delete_user_data acquired threading locks and ran rmtree in an async handler, freezing the event loop for every request whenever deletes contended. A plain def handler runs in FastAPI's threadpool where blocking is safe. Lock ordering (user lock, then store lock) is unchanged and consistent across threadpool workers. Per review feedback on #161. Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
Replace the bare RuntimeError raised by closed stores with StoreClosedError (shared in app/services/store_errors.py) and register an exception handler so in-flight requests losing the race against user deletion get a 410 instead of an unhandled 500. set()/delete() now check the flag before mutating so a refused write cannot leave a phantom record in memory. Per review feedback on #161. Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
A closed store kept serving the deleted user's records from memory to any captured reference. Dropping the dicts in close() (belt and braces with the closed flag) makes reads return empty rather than stale data; in-flight reads racing deletion were already best-effort, so nothing meaningful is lost. Per review feedback on #161. Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
The store write guard did not cover plain file writes. A trace awaiting its model call could outlive a DELETE /users/me: the tracer then ran mkdir(parents=True) and wrote the mask into a recreated dir, after which the store set() raised and the file was orphaned. The captured (now closed) store is the signal: trace_tools passes ensure_open into the tracer as a before_mask_write hook, and the upload, trace-mask and shared STL generation paths check it before writing into the user tree. Fresh requests for a deleted user id still recreate dirs by design. Regression test simulates a delete landing mid-trace and asserts the user dir and mask never reappear. Per review feedback on #161. Signed-off-by: Jason Madigan <jason@jasonmadigan.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
generate_stl) can no longer rewrite or serve the deleted user's data.get_stores/get_project_storeand the deletion sequence (evict, close, rmtree) serialise on a per-userthreading.Lock, closing the residual Evict _project_store_cache on user deletion #159 window where a store could be built from files mid-rmtree and cached with stale data.StoreClosedError, mapped to a 410 at the API layer, so requests losing the race fail cleanly instead of as unhandled 500s.StoreClosedErrorand nothing reappears on disk, including after dir recreation), store creation racing an event-gated slow rmtree, and a delete landing mid-trace (asserts the user dir and mask never reappear).Approach
The issue offered a per-user lock or a deleted-generation check in
_save. Neither alone closes the race, so this uses both halves, each for the window only it can cover:set()/delete()hold the store's existing lock for check + mutate + save, andclose()takes the same lock, giving a clean happens-before (write lands before rmtree, or the write sees closed and refuses).close()also drops the in-memory dicts, belt and braces with the flag, so a captured reference cannot read stale records either; in-flight reads racing deletion were already best-effort, so nothing meaningful is lost.Non-store file writes get the same treatment, with the captured (now closed) store as the deletion signal. The seconds-long window is a trace awaiting its model call: the tracer used to mkdir + write the mask when the call returned, silently recreating the deleted user's dir before the store
set()raised, orphaning the mask with no cleanup.trace_toolsnow passes the captured session store'sensure_openinto the tracer as abefore_mask_writehook that aborts the write, and the upload, trace-mask and shared STL generation paths check the captured store before writing into the user tree. Fresh requests for a deleted user id still recreate dirs by design.Refused writes raise
StoreClosedError(aRuntimeErrorsubclass shared inapp/services/store_errors.py), mapped to a 410 with a terse detail by an app-level exception handler. The delete handler itself is now a syncdef: it takes threading locks and runs rmtree, which belongs in FastAPI's threadpool, not on the event loop.Locks are never removed from
_user_locks(popping reintroduces races); the dict is bounded by user ids seen, fine for the single-process scope here. Multi-process remains #155.drawer_store.pyis untouched: nothing instantiates it.Test evidence
pytest(backend, main checkout venv): 298 passedmake lint-backend: passes (frontend untouched; worktree lacks node_modules for eslint)sessions.json/tools.json/bins.json/bin-projects.jsonreproduced before the fixbefore_mask_writehook disabled: the orphaned mask and recreated user dir reproduceCloses #160