Repository navigation
Description
DRAFT — godot_voxel shutdown deadlock (отдельный от #902) · не отправлено
Статус: драфт 30.09. Отправить после верификации cherry-pick 10d1a22
(workflow dwfrun-719be148), чтобы сослаться на свежее состояние форка.
Title: Shutdown deadlock: endlessly-requeued POSTPONED tasks keep wait_for_all_tasks's pickup loop alive (main thread parks in "Waiting for all tasks to be picked")
Body:
Hi! Separate from the crash we discussed in #902 (whose headless component your
10d1a22d already fixes — thank you; we cherry-picked it onto 595f52ee and verified:
the same workload that died 6/6 now survives 4/4), we keep hitting a shutdown
deadlock that the fix does not cover. It reproduces both headless and windowed, and also with
VoxelEngine.set_thread_count(1), so it is a main↔worker interaction rather than
worker concurrency.
Reproducer (minimal project from #902, GDScript-only):
https://gist.github.com/0-danielviktorovich-0/d014a383264c1d15f45d8276fd0235ed
VoxelTerrain + VoxelMesherBlocky + a GDScript hills generator, VoxelViewer
teleporting around (view_distance 384-512), then quit. Typical outcome: the game
prints the scene's final frame, then
WARNING: Waiting for all tasks to be picked is taking a long time
at: wait_for_all_tasks (util/tasks/threaded_task_runner.cpp:436)
and the process never exits (workers spin at ~340% CPU in one variant; in another
it later aborted with EXC_BAD_ACCESS at null+0x8 — report in the #902 gist:
Godot-2026-09-30-213018-minproj-quit-hang-crash.ips).
Symbolized sample of a hung process (in the gist as hang-sample.txt) shows:
- main thread:
ThreadedTaskRunner::wait_for_all_tasksat threaded_task_runner.cpp:424/433
(recursive_mutex /OS::delay_usec) — the first loop, the one that waits for
tasks to be picked; - workers:
thread_funcat threaded_task_runner.cpp:203 blocked in
recursive_mutex::lock, with sampling branches inside task sorting
(thread_func:248,SortArray<TaskItem>) and insideGenerateBlockTask::run_cpu_generation.
Suspected mechanism (from reading threaded_task_runner.cpp at 595f52e): a task
that returns STATUS_POSTPONED is re-queued into _spinning_tasks
(thread_func :358) and immediately re-picked on the next iteration (:192), so a
task waiting on a condition that teardown will never satisfy (e.g. neighboring block
data that stopped arriving) cycles forever. The first wait_for_all_tasks loop only
exits when _staged_tasks, _tasks and _spinning_tasks are simultaneously
empty — the cycling keeps _spinning_tasks non-empty and the main thread parks
indefinitely. The TODO How to cancel meshing tasks? at voxel_engine.cpp:162 looks
adjacent: there is no cancellation path for tasks whose dependencies become
unreachable at shutdown.
Ask: does a cancellation/timeout story exist for POSTPONED tasks at teardown?
We can test any patch on both the minimal project and our full-game harness (the
hang reproduces there too, e.g. after quit with a busy streaming queue at
view_distance 384).
Environment: godot_voxel 595f52e (+ your 10d1a22 cherry-picked locally),
Godot 4.7.2.stable.official, macOS 27 arm64.
Activity
Zylann commentedon Sep 30, 2026
Coming back from the other issue:
Are you using
VoxelGeneratorMultipassCB? Because this is the only one AFAIK that uses the postponing feature.I really hate the way the multipass generator turned out to be. This is one of the reasons. That "postponing" feature is something I came up with while making that generator and in retrospect I feel like it wasn't a good idea.
In a situation like this, that task is supposed to detect that it needs to be aborted, but somehow it doesnt, or can't.
But from the looks of it it appears you're not using this generator? So how is it possible that postponed tasks are coming up?
0-danielviktorovich-0 commentedon Sep 30, 2026
Thanks for the quick answer!
No, I don't use
VoxelGeneratorMultipassCB— the minimal project (and the full game) useVoxelGeneratorScript(pure GDScript hills) +VoxelTerrain+VoxelMesherBlocky. So you're right that postponing shouldn't be in play, and my POSTPONED hypothesis was likely a red herring — I apologize for the speculation.What my symbolized
sampleof the hung process actually shows (from the gist,hang-sample.txt):ThreadedTaskRunner::thread_func→GenerateBlockTask::run_cpu_generation(generate_block_task.cpp:143) — i.e. executing the GDScriptgenerate_blockon a worker thread — with sampling branches blocked inrecursive_mutex::lock(thread_func:203, the task-pickup mutex);wait_for_all_tasksfirst loop.New hypothesis that fits "no multipass": at quit, Godot tears down the script VM / language lock while a worker is still inside the GDScript
generate_blockcall. The worker then holds (or waits on) the script lock forever, never finishes its task, andwait_for_all_taskswaits for a task that can now never complete. That would also explain why it reproduces withset_thread_count(1)— one worker parked inside the script call is enough.If that's plausible, the actionable question for you is narrower: should
wait_for_all_tasks(or the VoxelTerrain shutdown path) guarantee that script-touching tasks are cancelled/completed BEFORE Godot's script server starts deinitializing? And is running aVoxelGeneratorScriptgenerate on a worker during shutdown a supported state at all in 1.0.1 / 595f52e?I can test any patch on the minimal project (hangs 3/3, ~always at quit) and the full game. One more data point: the same hang sometimes ends in
EXC_BAD_ACCESS null+0x8on the parked main thread (ips in the #902 gist) — consistent with teardown proceeding under the parked worker.Zylann commentedon Sep 30, 2026
Dunno if that's the problem either. But for sure I'd like the script server to deinitialize afterwards indeed, but I dont know how to ensure that. Singleton order is often a shitshow, I find. The fact you are allowed to write script generators at all pretty much requires support for the eventuality that one of them is pending if the game is shut down while tasks are underway.
Is that happening specifically when you use scripts? It's already really annoying to even expose this stuff to GDScript (notably due to lack of safety already mentionned in docs, and various performance issues).
Edit: from what I recall the
VoxelEnginesingleton is destroyed atMODULE_INITIALIZATION_LEVEL_SCENE, which I assume is before the script server already?0-danielviktorovich-0 commentedon Sep 30, 2026
Yes — it happens specifically with script generators. And your singleton-order suspicion matches what I see.
Decisive A/B from the minimal project (same seed, same camera path, same windowed run, only the generator swapped):
VoxelGeneratorFlat(native)VoxelGeneratorScript)wait_for_all_tasksFull game: same split (native-core paths exit clean; script-generator sessions hang at quit while a worker is parked inside
generate_block, per the sample in the gist).So: a pending script task at shutdown is exactly my trigger, which matches your "writing script generators at all requires support for the eventuality that one of them is pending at shutdown". Agreed the Godot-side guarantee is hard to make from a module (singleton deinit order being "a shitshow" is real); from my side the useful contract would be anything from "documented unsupported: script generator tasks do not survive shutdown" to a
wait_for_all_tasksvariant that force-completes/cancels script-backed tasks before release. I can test anything on the minimal project (hang is ~100% reproducible there).Also happy to move my generator off GDScript if that's the practical answer for production use — my measurements show the script generator is also my slowest streaming component anyway.
Zylann commentedon Sep 30, 2026
What gives then? How can the script server block the task from completing if it supposedly shuts down AFTER?
I mean, I don't know what else to do on my side. Having this issue with GDScript if quite a big pain in the ass and needs to be fixed somehow. IF that really is the problem though...
Because if that can't be fixed then anyone using GDScript in generators would have to implement stupidly annoying workaround to somehow force the game to "wait" when quitting, which is completely ad-hoc and unexpected burden.
Side question, am I talking to an AI? See #901 (comment)
0-danielviktorovich-0 commentedon Sep 30, 2026
Update with a deeper sample of the hung process (my symbolized rebuild, 5s
sample) — this changes the picture again, and I think for the better:A worker is stuck INSIDE
SortArray::introsort— spinning forever, not blocked.while the OTHER workers are all blocked on the tasks mutex at
threaded_task_runner.cpp:203(waiting for the spinner to release it), and main is inwait_for_all_tasks's first loop. This also matches the ~340% CPU spin I measured earlier.That's the classic failure mode of a comparator violating strict weak ordering:
introsortnever terminates.TaskComparatorcomparescached_priority(a float fromTaskPriorityDependency::evaluate). If any queued task evaluates to NaN (or an otherwise unordered value — +inf paired with NaN), every comparison involving it returnsfalse, the sort's invariant breaks, and introsort can recurse/loop indefinitely — while holding_tasks_mutex, which blocks every other worker at pickup (:203) and keeps_tasksnon-forever-nonempty forwait_for_all_tasks.This would explain:
set_thread_count(1)sometimes dodges it (fewer concurrent evaluations, different interleaving).Proposed check on your side: assert/log
Math::is_nan(cached_priority)in the pickup loop before sorting. On my side: I can rebuild my fork with a diagnostic that prints the offending task'sget_priority()inputs (viewer positions, drop distances) the moment a NaN is seen, and run the minimal project until it fires. If NaN is confirmed, the fix could be as small as sanitizing the priority inTaskItem(or making the comparator NaN-safe), plus finding which input produces it.Full sample is in the gist from #902 (adding
hang-deep-sample.txt). I'll run the NaN-diagnostic build and report the offending values.Zylann commentedon Sep 30, 2026
cached_priorityis not afloat. Some floats might be used when priorities are calculated, but after that it's all integers, including during sorting. Seestruct TaskPriority, that's what is compared.The sorting can take a while if there are many tasks. One can easily flood the system with tasks with a slow generator on a big enough terrain. But I assume that's not your case, right?
0-danielviktorovich-0 commentedon Sep 30, 2026
Correction to my NaN theory — withdrawn.
TaskPriorityis a union of fouruint8_tbands over auint32_t(whole), and the comparator iswhole < other.whole— strict weak ordering is guaranteed, so introsort cannot loop on NaN. Sorry for the noise.What the sample actually shows, stated more carefully: workers cycling in pickup (mutex at :203) and periodic full sorts at :248, main parked in
wait_for_all_tasksfirst loop, ~340% CPU sustained for many minutes. Withview_distance 512, the request set is on the order of hundreds of thousands of blocks, and with a GDScript generator each generate is roughly a millisecond — so "wait until work runs out" can mean hours of legitimate draining.So my current working theory is much more boring than a deadlock: this is not a hang at all — it's
wait_for_all_tasksfaithfully waiting for a backlog that would take hours to drain at shutdown, because nothing cancels the remaining requests when the tree is being freed.If that's right, the actionable question becomes: should the VoxelTerrain/VoxelEngine shutdown path cancel (drop) remaining queued tasks instead of waiting for them? The worker threads would finish their current task and exit cleanly, and quit would take seconds instead of hours. I've added a temporary queue-dynamics print (tasks/staged/spinning/completed every 2s from the pickup loop) to my fork to confirm the backlog shape — first numbers next update.
On "Is that happening specifically when you use scripts?": the trigger needs the slow generator to make the backlog drain-time explode, and yes, ours is a
VoxelGeneratorScript. With a nativeVoxelGeneratorFlatthe same backlog drains fast enough that quit completes — which is why it looked like a script-only issue.Zylann commentedon Sep 30, 2026
Hard judgement to make, Some might be important, and
wait_for_all_taskscan't really tell from its side.Something to note also,
GenerateBlockTask(the task that then runs the script generator) has anis_cancelledmethod that may returntruebefore the runner runs it. Inside it checks ifStreamDependency::validis still true. The destructor ofVoxelTerrainsetsStreamDependency::validtofalse. So... shouldn't generation tasks be skipped in the first place, already?But if despite this your setup ended up with enough tasks queued for hours, the program might as well be killed, and then I would question the way you implemented your generator, or the way you setup the terrain, which needs fixing in the first place.
0-danielviktorovich-0 commentedon Sep 30, 2026
Queue-drain hypothesis confirmed by live measurement (diagnostic build with the queue print is still gated on my disk budget, so here's the cheaper proof):
10-minute CPU/RSS trace of the hung process, 30s resolution:
A deadlock would sit at ~0% CPU. Instead, ten minutes in, the process is sustaining 330–577% CPU and actively allocating/freeing chunk memory — the workers are genuinely grinding through the backlog.
wait_for_all_tasksisn't stuck; it's faithfully waiting for a generate queue that (atview_distance 384with a per-block-millisecond GDScript generator) needs hours.So the minimal-project "hang" is: quit → wait_for_all_tasks → drain of the full view-distance backlog with no cancellation path. In the full game this is invisible at view_distance 192 with fast regions, and fatal at 384+ with a slow generator.
The actionable question stays as in my previous comment: would you accept a patch/option that drops (instead of waits for) remaining queued tasks on shutdown — e.g.
VoxelTerrainnotifying the task runner to cancel-and-exit-wait, so workers finish only their current task? I can test any shape of it on the minimal project (hang is ~100% reproducible) and the full game.Queue-print numbers from the diagnostic build will follow once my disk budget frees up — but the CPU/RSS trace above already settles "deadlock vs drain".
Zylann commentedon Sep 30, 2026
See #903 (comment)
Also
0-danielviktorovich-0 commentedon Sep 30, 2026
That closes the loop — and it points at the
#189workaround in your ownVoxelEngine::~VoxelEngine():My hang sequence, reconciled with that:
VoxelTerraindtor setsvalid=false(voxel_terrain.cpp:106-107) — pending tasks will be skipped from now on;GenerateBlockTaskalready dequeued is inside a GDScriptgenerate_blockcall;wait_and_clear_all_tasks(false)(the The order of destruction of GDScriptLanguage::singleton prevents from freeing any script fromunregister_voxel_types()#189 workaround) then waits for that worker — parked in a script call while the script language goes down. It never returns → main parked, workers at 330-576% CPU churning the remaining native queue.And yes to your question — it's deterministic with scripts, same project, same camera path:
VoxelGeneratorFlat(native)VoxelGeneratorScript)Practical takeaway for me: the generator must be native — I'm moving to a Rust core regardless. For upstream, if you ever want to harden quit: track "script-backed tasks in flight" and have the SceneTree-teardown clear force-drop script ones (they re-queue on the next run anyway). I can test any shape on the minimal project — hang is ~100% reproducible there.
Zylann commentedon Sep 30, 2026
The expected flow would be: when
VoxelTerrainis destroyed, it setsStreamingDependency::validtofalse. Then, it unregisters fromVoxelEngine, which in turn sees there are no volumes left, and triggers the workaround callingwait_and_clear_all_tasksearly. It hangs there until done. Even if there was thousands of tasks pending, only one per thread might be running already. And these will finish because at this point the main thread is still within the destructor of a node, far from script server teardown. Theis_cancelledof pending tasks will returnfalse(sinceStreamingDependency::validis false) and they won't run. That was the point of the workaround. So... what is going wrong in there?0-danielviktorovich-0 commentedon Sep 30, 2026
That expected flow is exactly why my numbers are confusing — and why I'm building the queue-print diagnostics now. Facts from the minimal project that don't fit "pending tasks get skipped, running ones finish in ms":
get_tree().quit(), the process stays alive 10+ minutes at 330-576% CPU (30s-resolution trace in the gist from SIGSEGV (null+0x50) on mesh worker thread — godot_voxel 1.6 (595f52ee), macOS arm64 editor, symbolized stack + 6 identical crash reports #902);sampleof that state shows workers inside pickup (:203mutex) and the periodic full sort (:248), not insidegenerate_block— consistent with your expected flow (running GDScript generates are ms each), inconsistent with them being skipped or finishing;wait_for_all_tasksfirst loop still sees non-empty queues after 10 minutes.So either
valid=falseisn't reaching the pendingGenerateBlockTasks before the earlywait_and_clear_all_tasksfires (ordering betweenVoxelTerraindestruction and the volume unregister?), or the queue is being refilled during the wait, or the pickup/sort path itself is spinning on a huge backlog. The queue print (tasks/staged/spinning/completed every 2s) will discriminate between these — numbers in the next update once my build window opens.No action needed from you until then — the diagnostic build is on my side.
Zylann commentedon Sep 30, 2026
Long shot, maybe try this
0-danielviktorovich-0 commentedon Sep 30, 2026
Applied as
1fe6d898on my fork — thank you. Two honest notes:The live test of this patch runs in my next build window (diagnostic build with the queue print is being prepared; numbers in the next update). I'll report whether it changes the hang behavior.
Scope check: my deepest hang sample shows workers spinning in pickup (
:203) and the periodic sort (:248) while a GenerateBlockTask (the GDScript generator, not a mesh task) backlog drains — and in the minimal project the generator is the only scripted component. Your patch fixes a real shadowing bug for MeshBlockTask cancellation (a valid token shadowing terrain destruction), which I applied as independently correct — but the generate-side hang may need the same "check dependency validity first" treatment inGenerateBlockTask::is_cancelledif it has a token branch too. I'll verify both with the queue-print numbers.0-danielviktorovich-0 commentedon Oct 1, 2026
Answering your "what is going wrong" — with a correction to my own speculation first: the NaN/introsort theory is withdrawn (
TaskPriorityis uint32 bands, comparator is fine). Here's where things actually stand:Facts from the minimal project after quit:
:203) and the periodic sort (:248) — your expected flow says pending tasks should be skipped and running ones finish in ms;wait_for_all_tasksfirst loop);VoxelGeneratorScript,valid=falseis set inVoxelTerrain's destructor.Current reading: this may not be a hang at all — at
view_distance 384the request set is huge, and with a per-block-millisecond GDScript generator, draining it honestly takes hours. No deadlock needed to explain 340% CPU.Question back: should the shutdown path drop queued tasks instead of waiting for them? If GDScript generation at shutdown is unsupported anyway, I'll simply drop script generators from my production path (moving to a native core regardless) — and this issue becomes "won't fix for script generators" with my thanks for the diagnosis help.
Diagnostic build with a queue print (tasks/staged/spinning/completed every 2s) is compiling on my side — real numbers next update.
0-danielviktorovich-0 commentedon Oct 1, 2026
Queue-print numbers are in — the drain hypothesis is confirmed with live data.
Diagnostic build (my fork: your is_cancelled suggestion + T1 inversion fix + queue print every 2s), minimal project,
view_distance 384, GDScript generator:The queue is ~2.4 million tasks and GROWING ~34k tasks/second — each generated block enqueues more follow-up requests (neighbors, meshing, LOD) than the drain consumes (~11k completed/second over the window). At
view_distance 384the full spherical request set is in the millions; workers at 330-576% CPU can never converge.wait_for_all_tasksthen waits for a backlog that grows faster than it drains. No deadlock, no script-teardown issue — pure queue arithmetic.This reframes the original report: the "hang" is
get_tree().quit()→wait_for_all_tasks→ unbounded backlog with zero cancellation.VoxelTerraindestruction only setsvalid=falseafter the wait completes, so theis_cancelledskip never gets a chance to apply during the drain — and with the T1 inversion fix applied, it would only matter for tasks queued before terrain destruction, not this 2.4M-task live set.Practical status on my side: production moves to a native generator (GDScript generation is my slowest component anyway). If you want, I can also test a candidate patch that makes
wait_for_all_tasksdrop pending tasks when the scene tree is shutting down — say the word and I'll run it through the minimal repro.0-danielviktorovich-0 commentedon Oct 1, 2026
Live queue-print numbers from the diagnostic build (my fork: your
is_cancelledsuggestion + the T1 inversion fix + queue print every 2s), minimal project,view_distance 384, GDScript generator, second run 22s window:The queue grows from 2.7M to 4.0M tasks in 22 seconds (+55k/s inflow) while the drain consumes ~7k/s. Each completed generate/mesh block enqueues more follow-up work (neighbors, meshing, LOD boundaries) than a single completion removes — inflow structurally exceeds outflow at this view distance with a script generator.
wait_for_all_taskscannot converge; the 330-576% CPU is the pool honestly churning through a set that regenerates itself.So the shutdown path isn't broken per se — it's faithfully waiting for a queue whose inflow exceeds its outflow by ~8×. That makes the fix unambiguous: queued tasks must be cancellable/droppable at shutdown (your
is_cancelled-skip already works for not-yet-started tasks oncevalid=false— the missing piece is makingwait_for_all_tasksitself bounded during SceneTree teardown).My production path moves to a native generator, which shrinks the inflow by orders of magnitude. If you'd still like a defensive fix upstream (bounded shutdown wait or drop-on-teardown), I can test it on the minimal project the same day.
0-danielviktorovich-0 commentedon Oct 1, 2026
Shutdown acceptance test passes — clean exit in seconds, no drop-on-teardown patch needed on my side.
Same minimal project, same diagnostic build (your
is_cancelledsuggestion + the generate-side inversion fix + queue print),view_distance 384, GDScript generator,--headless:At quit the queue held 4.35M tasks; within ~2s of SceneTree teardown the pickup loop cancelled all of them through
is_cancelled()(cancelled items are counted in the completed counter in my diag print, hence the jump) andwait_for_all_tasksconverged immediately.A/B against the pre-fix build (same project, same view distance): after quit it sustained 330-576% CPU for 10+ minutes and was still climbing. The only operative change between the two builds is the cancellation behavior, so the validity-first
is_cancelledis what makes shutdown bounded (~2.2M tasks/second skip rate here).So following your guidance: I did not implement
stop_accepting_and_drop(), and I don't delete tasks insideScriptInstance::call— the existing skip path suffices at this scale. Caveat: this run is headless; the pre-fix hang reproduced windowed as well, so I expect the same post-fix result there, but I haven't re-measured windowed yet.This closes the shutdown issue for my production path — which moves to a native generator anyway, shrinking the inflow side by orders of magnitude. Thank you for the diagnosis help.
Zylann commentedon Oct 1, 2026
Sorry if that sounds rude but I got tired of the infodump so I stopped reading, it's too much for not enough value (I'm assuming you're AI as you didnt answer that question, twice).
I made my own simple test project and identified the issue in 045ff93