Repository navigation
TrainerRank reusable-cache admission can leave insufficient physical CUDA memory for backward library allocations #870
Description
September 25, 2026 — #888 is merged; general physical-headroom safety remains open.
PR #888 merged as e2e8969ff1f64151ae613abf0050e236ee77e35e, alongside the separate #898 accounting correction. Earlier dated held statements below are historical. The shared recovery budget does not guarantee total training slowdown or enough memory for every backward.
The follow-up is now draft PR #972: one optional budgeted cache-recovery attempt before accepting an internal split, followed by normal fresh admission. Three independent source reviews cleared f93b6d85338b62c52c6e0e2e66b95ceddd8b7d24; significant packing/admission behavior remains held for Brad. Quality CI found type-narrowing/test-annotation errors, with an isolated correction underway. Required hosted GPU CI failed during acquisition of two H200s before tests. A separate exact-head local five-command GPU suite is running; it does not waive the hosted gate.
An independently audited six-forward/six-backward model diagnostic on the predecessor with identical ordinary-path runtime completed without optimizer updates. It made two actual cache releases, freeing approximately 30 GiB each; release calls took 0.178/0.261 seconds and full recovery episodes 0.438/0.521 seconds. The second release passed the existing budget, but 93.4% of its work denominator was warmup/compile-associated. This is not steady-state overhead qualification, a throughput comparison, or numerical equivalence. The control stopped at its strict layout guard when admission selected a split. Audit receipt: /home/brad/.local/share/schulman/recovery-candidate-native-independent-audit-20260925/RECEIPT.json (188307fd). The final-head difference clears a transient plan on the separate oversized-override return; it has focused CPU lifetime coverage.
A two-GPU real-NCCL fixture independently passed four injected control-flow cases, including an empty peer, peer veto and local release error. It used synthetic memory accounting and release shims; it does not qualify distributed real allocator behavior. Receipt: /home/brad/.local/share/schulman/krennic-packed-boundary-20260925/recovery-nccl-native-audit-v1/REVIEW-RECEIPT.json (c4801cb6). Both completed diagnostics are physically closed. No new whole-model safety claim or issue closure follows.
Earlier dated history is preserved below.
September 23, 2026 — current recovery candidate qualified in additional bounded cases; still open and held.
Current #888 is 3b0c6d4ef8972aeba0441ec89756b70c8f227624. Three source reviews and quality/two-H200 CI are clear. It shares the measured recovery budget with gradient handoff and credits completed backward work. The 3% physical reserve is a soft trigger, and the 5% ledger does not guarantee total training slowdown or sufficient memory for every backward. Fable's additional review holds the policy pending explicit acceptance or useful targeted timing evidence: later allocator regrowth is not separately charged. No current-head merge is authorized by these results.
Earlier missing current-source checks are now complete at retained source 67401664: a positive-pressure release followed by backward/update, and grouped save with exact saved-versus-fresh adapter/master/optimizer/custom state followed by another update. The complete eight-pair/flat16 workload also passed on combined f80dba32. The new 053-shaped case passed one joint update on one H200, but retains manual caller cache release and is therefore not automatic-policy qualification.
The current-source same-H200 ON/OFF pair is complete: both arms finished six warmup and six measured updates, with original tasks/managed waits 0 and all 12 distinct known native identities/eight groups closed. ON's first failed close check and subsequent natural retirement are preserved. The unchanged primary comparator is INCONCLUSIVE because warmup call/partition records, measured partitions and starting reserved-cache bytes differed. A separate, predeclared full-work observation is 518.711 seconds enabled versus 504.801 disabled: +13.910 seconds (+2.756%). Both completed matching logical requests, output shapes and updates from the same saved initial state, but measured-start parameters differed after normal-policy warmup. ON recorded one proactive release adding 27,881,635,840 bytes of physical free memory; OFF skipped 35 proactive calls and had one observed allocator retry. These facts do not isolate regrowth cost or establish a universal five-percent slowdown bound. Independent retained-result review supports this scoped interpretation; no extra benchmark, policy adoption or merge follows. Final compact evidence: 888-current3b0-onoff-result-20260923/ under the receipt directory below.
Receipt directory: /home/brad/.local/share/schulman/art848-resume-20260921-root/. Completed earlier checks: 888-current674-pressure-final-root-acceptance.json (ce5715e5), 888-current674-grouped-save-fresh-load-root-acceptance.json (7bed2fd1), and combined-full8-native-v6-final-root-acceptance.json (47cb19ea). Current ON result: /var/tmp/art888-local-onoff-ON-20260923-halley-v1/actor-result.json (a27dafca); additive known-process closure cbf074b6. General backward safety, representative performance and repeated policy behavior remain broader than these bounded checks.
Earlier dated history follows; its old heads and pending-work statements are superseded above.
September 22, 2026 update — still open; #888 remains draft/held.
Current #888 is 5a4ea977552fa9b337ca65d8e3fabfa22fd1e887; exact-head quality and hosted two-H200 CI passed. It uses the shared budget for gradient-handoff cache release and credits completed ordinary backward work through a bounded observer. The 3% physical reserve is a soft trigger; the 5% budget is not a guarantee of total training slowdown or enough memory for arbitrary backwards.
The frozen d129f724 candidate completed six warmups plus six measured optimizer updates: 803.215 seconds warmup and 511.498 seconds measured work. Its matched main attempt had stopped before measured updates, so no baseline ratio follows. The corrected compile-scope reader still reports UNKNOWN due to unscoped warmup events; no compilation events were observed during the measured interval, which does not prove a fully stable compiled workload. All resources from these runs are closed.
The current public head passed fresh graph and installed-CPU qualification for grouped save/fresh load. Its GPU attempt failed before cloning or trainer creation: the private guard required raw complete=True, whereas grouped durability is established through the committed group authority. The original attempt did not retain donor metadata, so its exact failing predicate remains unknown. Native exit1 and complete known-resource cleanup are preserved; no OOM, training update or save result is inferred. A narrow guard correction and its standalone/prior-group cases are under review. Separate matched current-main/current-candidate performance preparation is underway with identical eight-pair normalization and two four-pair gradient calls.
The earlier successful grouped save/load and positive/repeat-pressure episodes remain historical-source evidence, not qualification of the new backward observer. Still needed: current-source grouped save/fresh reload, matched full-model timing, and scoped positive/repeated budget behavior. General #870 remains broader than these tests. Durable current receipts: art848-resume-20260921-root/888-final-d129-candidate-only-root-success-closure.json (560743bf), 888-H1-grouped-native-failed-guard-root-closure.json (4fe8a10d) and 888-main45-H1-four-plus-four-graph-root-release.json (b6b38c11). No merge, deployment or tolerance change.
Earlier dated history is retained below; this update supersedes old head and pending-run statements.
Current status — September 21, 2026, 6:35 AM Mountain: unresolved; PR #888 remains held.
- Current draft Share budgeted CUDA cache recovery with gradient handoff #888 is
b90678708bdc22672f5a4ab66a6f23c4a5c5e641. Its budgeted release policy is not merged. The current main/candidate CPU image prerequisites passed; representative warmed throughput, grouped save and fresh reload remain unqualified. - The first fresh same-H200 pair attempt filled its fixed model cache, then stopped before GPU keeper allocation: the Kubernetes all-pods response exceeded the runner's artificial 2 MiB raw-response cap. The assertion preceded body/wait persistence, so the second response's exact size and OS status are unknown. No measured epoch or save/reload occurred. All cache/fill resource UIDs and known host groups are independently/root closed; no live cache adoption is implied.
- A narrow private runner correction preserves the completed query's wait/count/hash evidence before refusal and permits raw combined output below 64 MiB; the compact summary retains its 2 MiB limit. The same two GETs/timeouts/selection remain. Source review and four old-red/boundary/error CPU controls pass. Fresh final composition precedes another finite attempt; no production behavior change is bundled with this runner fix.
- A separate positive-pressure diagnostic is being integrated for a real release opportunity. It will remain distinct from the normal warmed pair and cannot substitute for total-overhead measurement. No new native pressure result is claimed.
Owner: Schulman and existing delegates. #848 estimator/admission and #901/#902 numerical questions remain separate. Actual host cache stays counted. Durable report: /home/brad/.local/share/schulman/art848-resume-20260921-root/PROGRESS.md.
Earlier status/history below is preserved; current statements above supersede old heads and pending-run descriptions.
Current status — September 18, 2026, late evening Mountain: partially addressed; still open.
PR #900 is merged. Current held PR #888 is at f557bc4e60f7b38bead1cc0efd7d655e2bc7df69; its low-physical-headroom backward handoff now uses the same measured cache-release budget, rather than an unconditional release rule. Three exact-head source reviews and CI, including two-H200 checks, are clear. Significant behavioral adoption still needs Brad's decision.
A full-model candidate run completed three real optimizer updates (18 backwards), followed by a separate pressure diagnostic with one automatic handoff release and one successful backward, with no optimizer update during that pressure diagnostic. It then successfully published the grouped generator/auxiliary checkpoint. The original run nevertheless exited 1: after trainer close returned, its actor-absence check timed out before fresh reload began. This failure is retained and is not classified as a backend or zombie-process cause. Exact owned resources were independently retired afterward.
A distinct load-only continuation is now running on one free H200: art848ws-07a21878, launched at 11:04:39 PM Mountain. It uses protocol-8 Caladan with the original candidate ART source, loads the exact saved generator:v1/auxiliary:v1 group from art848ws-b140393f, compares complete canonical optimizer/custom state, then attempts exactly one update and real cleanup. Prior training, pressure and remote grouped publication are not repeated. Its actual worker-image and restored-helper CPU checks passed; native saved-state equality and successful completion are not yet claimed.
The observed release cost is not a total-training 5% overhead bound. Repeat-policy instrumentation itself contributes to measured work/cost, and process-wide cache effects remain relevant. A separate source proposal can measure two pressure episodes and intervening updates, but is not yet run and cannot by itself establish representative throughput or arbitrary backward safety. Keep this issue open separately from #848 and numerical variability #902.
Evidence: /var/tmp/art888-load-only-candidate-image-20260919-root/cpu-root-acceptance.json (041f584e), exact native authority gpu-root-admission-v2.json (d4fb5fd6) and gpu-live-identities.json (e16daabc). Original failed producer evidence remains /var/tmp/art888-native-sequence-20260919-backward-n2_crwep/operation/. No merge or deployment occurred in this follow-up.
Earlier status/history, preserved; current statements above supersede old head and pending-candidate descriptions:
Current status — September 17, 2026: partially addressed; still open.
Brad approved PR #900, merged as c009557e2f2488a83551d29c3253b708edef322c. Admission now uses physical free CUDA memory and can release unused allocator cache under a measured cost budget. The scoped two-H200 qualification and source/CI checks passed before merge. This does not establish full-training overhead, sufficient memory throughout arbitrary backwards, or adoption by every Caladan experiment/image.
The earlier PR #888 is a separate, still-held proposal to release cache at a low-headroom handoff to backward. It is not adopted by #900, and its process-wide effects and throughput tradeoffs remain relevant. It should not be merged or relabeled as superseded without reassessing that remaining boundary against the current admission policy.
Active work with #848: qualify the current code on the retained workload and resolve underestimated backward demand. Keep the library-allocation problem separate from the repeated-gradient variability tracked by #902. All recent completed diagnostic resources are cleaned up. Representative end-to-end overhead and broader backward-memory safety remain unqualified.
The historical September 12 status below is preserved; its description of the then-current candidate is superseded by this update.
September 12 update: the earlier statement that conditional/full-step and multi-device behavior are entirely unqualified is superseded by bounded native evidence. Three complete updates on one H200 exercised four actual conditional releases; a separate two-H200 run verified that selected-device release also trims the other device's unused cache while preserving the checked live values/gradients and caller device/streams. All resources were independently retired. Exact retained evidence and limits: /home/brad/.local/share/schulman/art870-two-device-gap-inventory-20260912-capture/REPORT.md.
ART #888 remains OPEN/DRAFT at1763a46ac9b4f465e38440baf8cc3e7922da69a3, source-correctness reviews and CI green, adoption HELD. The 3% trigger is best effort, not a guaranteed reserve. Process-wide cache retirement/waits are significant behavior requiring Brad's decision. Representative throughput/refill costs, arbitrary concurrent users, and broader distributed/negative cases remain unqualified. Do not repeat the completed two-device witness merely to restate the already observed process-wide effect.
Earlier record (preserved; status above is current):
Status reconciliation — September 11, 2026 (Schulman)
Confirmed allocation failure; local mitigation only. Unconditional cache release let the pinned first-backward diagnostic finish, but the proposed conditional trigger, complete optimizer behavior and multi-device side effects remain unqualified. torch.cuda.empty_cache can trim caches across devices in the process; a selected-device context does not isolate that effect.
Planned lane: Schulman and subagents, queued with #848/#256. Complete the conditional/multi-device and full-step qualification before proposing a generic fix. The held local patch is not merged, does not modify art.megatron, and is not covered by a no-significant-behavior assumption.
Historical report (preserved):
Type: confirmed backward allocation failure with a controlled experiment-local mitigation; generic policy remains unresolved.
Agent owner: Schulman, related to #848.
A rank-1 Qwen3.6-35B-A3B eight-pair dynamics batch on one H200 passed its forwards, then failed in micro_batch_loss.backward with CUBLAS_STATUS_ALLOC_FAILED from cublasCreate(handle). GeneratorExit was secondary iterator teardown. The exact internal cuBLAS request size was not measured.
At the first gradient admission, live allocated/reserved memory was about 74.662/147.394 decimal GB with only 0.034734 GB physically free. The planner treated reusable allocator slack as available; its 3% capacity subtraction did not physically reserve CUDA memory for external library allocations. At the caller boundary, an unconditional torch.cuda.empty_cache() released 42.754638 GB without changing live allocation. The matched release-only replay completed all five backward waves and the optimizer; the observe-only replay failed. Pre-intervention plans/budgets and corresponding forward counters matched exactly.
The qualified mitigation is experiment-local cache release at the start of every gradient caller, after receiving its microbatch. It is not equivalent to a conditional 3% policy, a pre-yield trainer hook, or handle warmup. Later nonzero-generator-gradient updates also succeeded, but larger traces and full validation remain unqualified. Cold forward estimate underprediction is a separate problem.
Investigate a bounded generic physical-headroom/library-allocation contract with mixed-grad plans, device/stream behavior and normal downstream workloads. Prefer trainer_rank-local work; no art.megatron change is proposed. Do not silently globalize the workaround based on one fixture.
Evidence: /home/brad/.local/share/schulman/retail49-memory-failure-20260909/MORNING-ALLOCATOR.md and replay-{observe,release}-comparison/. Local Caladan mitigation 431e424 and timing-only follow-up 177a9fce. Raw failures and cleanup receipts are preserved.
Activity
bradhilton commentedon Sep 9, 2026
The new three-arm cold eight-pair comparison is complete and all owned resources are independently absent. Observe-only again fails at first backward with cuBLAS handle allocation, with41,615,360 bytes physically free. Both caller-release and trainer pre-yield release complete all five backwards plus optimizer with matching finite metrics and byte-identical auxiliary head outputs; first G gradient is zero by the initialized objective.
Both placements release42,754,637,824 bytes of reserved cache and raise physical free memory to about42.8GB. There is no demonstrated pre-yield advantage on this fixture, and the tested endpoints do not determine a safe generic threshold or establish a general library-allocation contract.
Terminal harness failures remain preserved: the caller arm completes its update, then a standard JSON boundary rejects the cal.run dict proxy; the pre-yield arm completes, then trainer.close exceeds60s. A CPU reproduction shows explicit dict(result) is appropriate at that JSON serialization boundary. External UID/service/process cleanup succeeds for all three arms. These are not clean exit0 runs.
Evidence:
/home/brad/.local/share/schulman/art-headroom-arms-20260909d/REPORT.md, SHA256d11ac00d7bdce959ef4a56636c20935bd9aff7317ec2753528097aea262a1831; sealed evidence manifest35f88e971b5daa8092782458fe793cc3316ab8ad80a9c116e1fa9271fc9d5d2a. The existing049 caller release remains in main. Generic trainer_rank policy remains unresolved; no art.megatron change or new launch.bradhilton commentedon Sep 10, 2026
The new bounded V7 observe run reached the exact first backward and preserved a finalized13,362,126-byte Nsight report (SHA25673948f6b4d55da54da4fcfe42be029a8bcf11623246b846a0c197750a7985001). Independent read-only SQLite analysis finds a concrete allocation failure: cudaMalloc_v3020 return2, correlation1293703, wholly inside cublasCreate_v2 on the same worker thread. The owning process is actorPID9568; the worker is TID37872, pt_autograd_0. It falls50.294ms after admission3's backward_start and5.575ms before its RuntimeError boundary. The main telemetry thread9568 is distinct; the association uses exact process namespace, same-worker CUDA/cuBLAS nesting and checked monotonic-clock alignment, not an invented CUDA admission label.
Admission3 reported required7,976,316,880bytes against available68,242,798,285bytes. At handoff, PyTorch allocated92,197,523,968bytes/reserved147,394,134,016bytes while physical CUDA free was14,352,384bytes. No backward completed. This supports physical allocation pressure during cuBLAS creation on this exact fixture, beyond the exception classifier alone. The failed request size, callchain and cuBLAS return-status field are absent from this export; some CUDA/NVTX events may be missing, and hardware tracing fell back to software. The exported memory table has allocation records only, so no complete allocation/free ledger or generic safe-reserve threshold is claimed.
Report
/home/brad/.local/share/schulman/art870-v7-trace-independent-20260910/REPORT.mdSHA252165e49246d0f8a613e74705ef87090c5857201749ae90d1693c7664922cc5; manifest23e50cf1ecd73e69edf4680c5f3ec86cf4263a76fa7040dcf159026519e0ebf1. All10 original/copy hashes, SQLite integrity, owner-token/transfer and timestamp/identity joins pass. No live-resource operation was performed for this analysis. A same-source release-mode intervention is the next bounded comparison under root registration; it would qualify that intervention on this fixture, not establish a general reserve policy.bradhilton commentedon Sep 10, 2026
#870 release-d: per-file trace cap censored the run
The original retained companion exception is
Trace file byte budget exceeded, fromsnapshot()'s regular-filest_size > max_file_bytescheck.It is not a CUDA OOM, an inferred safety margin, or a 90% threshold. The configured
per-file cap is 1,073,741,824 bytes. The exact offending relative path and size
were not included in the failure response, so they cannot be reconstructed here.
trace-guard.jsonis the last successful poll at 3:42:33 AM Mountain:1,027,608,576 largest-file bytes, 1,329,111,040 aggregate accounted bytes and
1,071 entries. The next poll threw before either observation receipt was updated.
The successful snapshot's remaining space does not describe the failing poll.
The original companion traceback joins the actual launched V9 source through
companion.py:194 → trace_guard.check:470 → remote_snapshot:428 → snapshot.Identity: source manifest
0f89ce6047d3436d26f0194a8cdce0fb07d760c0aff048c0528ce773e4f5bc84,release mode, root registration260707, exact pod UID
07fb3420-0c0d-406b-a968-9e1c4d05bbbe, actor PID6986, ART7ea4db7b implementation044c9953…, Caladan99441ee1 and immutable ecf9 image. The full input manifestrecords original paths, sizes and hashes without copying raw driver logs or
touching the frozen run. Uploaded/runtime guard bytes equal the launched source.
Phase evidence is limited:
Receipt times above are durable file timestamps where no event timestamp exists;
they are not precise actor-operation timing.
kernel.jsonis written immediatelybefore the queued measurement await. It proves the driver reached that boundary,
not that the actor entered or finished the function. No retained local record
proves prepass completion,
capture_start, gradient handoff/cache release, orbackward start/completion. No
first-backward.json, failure/closing terminal orcompleted entrypoint receipt exists. Cicero confirms this V9 composition has
terminal-only actor telemetry transfer; no separate actor capture is owned by
him. An unrelated successful observe trace cannot fill this attempt's missing
phase join. No model OOM is present in the retained evidence; its absence is not
proof of what the forcibly stopped actor was doing.
The retained canonical cleanup receipt reports no remaining pods, services or
processes,
sky_down_exit=0, and no cleanup errors. Root independently owns thatresource verification. This auditor made no Kubernetes, actor, queue or resource
calls and did not independently relaunch/revalidate a live object.
An unchanged-source CPU witness uses tiny owned scratch files and mocked GETs.
A 900-byte file passes a 1,024-byte cap; growth to 1,025 bytes raises the exact
error and leaves the previous durable guard receipt byte-identical. The scratch
was removed. It demonstrates the stale-success-receipt behavior without creating
a GPU workload, large file, Kubernetes object or source patch.
Smallest defensible next counterfactual
First make the existing bounded observation path survive another censor:
persist primitive collector events incrementally under the exact owner, and have
the existing companion read bounded summaries. At a file-budget failure, retain
the owned relative path, size, cap, partial inventory and failure timestamp
before terminating. Preserve error identity and all caps, UID/birth checks,
deadlines and canonical cleanup. This needs no new actor queue or supervisor.
The existing collector already limits records to256 and each record to16KiB;
any incremental response must also fit the monitor's64KiB response bound.
Then consider a new paired observe/release capture range from native gradient
handoff through the first backward, keeping the 1GiB/2GiB caps and existing
CUDA/cuBLAS trace switches. Start profiling in a source-gated instance wrapper
at entry to the existing
_test_gradient_handoff_cache, before itsbeforememory snapshot. Starting inside
_telemetry_phasewould be too late: thatsnapshot already exists, and profiler startup could contaminate the reported
before/after cache-release difference. Stop after first backward under the
existing ownership/error handling. Prepass and gradient forward still execute
unchanged, and original candidate admission remains captured; their detailed
CUDA trace would be excluded.
This is a scope-reduction hypothesis, not proof it will fit: the failed file's
type/contents are unknown and backward alone may still exceed the cap. It requires
CPU ordering/source/ownership/partial-failure tests and a fresh registered pair
with matched input/source/seed and actual admission/layout/state checks. A new
shorter-range release cannot be treated as an identical-instrumentation pair
with old observe-e. No optimizer, publication, arbitrary warmup or cap increase
is needed for the proposed diagnostic.
Dropping
--cuda-trace-all-apis=trueis another possible reduction, but not yet acoverage proof for the required allocation/library APIs. NVIDIA documents added
overhead from tracing every CUDA API and from memory-usage collection; exact
staged CLI help confirms those flags. Prefer the narrower range first, retaining
the target release/backward evidence. NVIDIA Nsight Systems User Guide
No implementation or retry is authorized by this report. Root retains scope,
independent review, source sealing, capacity and launch decisions.
bradhilton commentedon Sep 10, 2026
A separate prepass allocation failure is now documented for follow-up under the physical-headroom investigation. Fresh residual attempt J completed both no-grad forwards and all eight captured/tensorized trajectories, then failed in the unchanged
rank.dp_reduce(statistics)before any gradient forward or successful update. NCCL reportsFailed to CUDA calloc async 24 bytes; this message alone does not establish CUDA OOM, the failed allocation’s underlying cause, or a generic memory-policy defect.The preceding caller boundary records 75,600,376,320 bytes allocated, 147,463,340,032 reserved and 381,353,984 physically free. The larger admitted prepass used 228,142 packed / 928,901 logical tokens across seven pairs and 79 requests; its original required/available check was 68,177,016,880 / 69,433,447,117 bytes. Measured incremental forward peak was 46,946,880,000 bytes, below that estimate. These measurements do not demonstrate forward underprediction. Caller-owned last-microbatch tensors were still live.
The evidence is frozen at
/home/brad/.local/share/schulman/research-j-prepass-failure-20260910-harvey, manifest01d2dff0a5bf13a8b393317858309e0f3d6272091c170c11e83641e869f3360b. Exact run/pod/source/queue joins, original checks, all raw captures and errors remain private. Root independently verified the three process births/groups, pod and two service UIDs, and Sky owner absent after cleanup.A fresh, unarmed local L candidate adds exactly one
torch.cuda.empty_cache()immediately before the unchanged reduction. It retains all caller tensors and collectives and restores byte-for-byte to J after deleting that line. Independent review is in progress; 22 root CPU source/native/binder controls pass. It is an unqualified local mitigation, not a generic ART change. Fresh generated inputs will differ, so a successful future L would not be an exact replay of J. The cuBLAS first-backward trace and its separate observe/release experiments remain distinct evidence.bradhilton commentedon Sep 10, 2026
Schulman, September 10 morning update. This issue remains open.
The completed V11 observe-g trace joins a failed
cudaMalloc_v3020return 2 insidecublasCreate_v2on the same actor and backward thread. The full 3,074,756-byte Nsight report has SHA2563229d2f4358fde4769f930234f0d45968dba38891ac407fe0d634a2478433411; all thirteen primitive events were returned. Original admission: predicted 7,976,316,880 bytes against 68,242,798,285 available. Handoff: 92,197,523,968 allocated, 147,394,134,016 reserved, only 14,352,384 physically free. The failed allocation's requested byte count is unknown. The trace records an early failure and does not qualify the five-second capture-stop path.The earlier cache-release arm freed 42,754,637,824 bytes of reserved memory with allocated tensors unchanged, but was censored by trace/disk guards before any backward-end result. That is neither a successful backward nor an OOM result. Same-source V11 release-f then correctly refused before driver launch because free disk was below its unchanged 23-GiB starting requirement. Release-g was registered but never granted/launched; release-h remains unarmed. By the time staging and disk gates cleared, the remaining work window was shorter than the observed V11 launch-to-backward time of 610.86 seconds, before accounting for release completion and drain. This empirical timing is not a universal minimum. No deadline or budget was relaxed.
All allocated attempts have independent exact pod/service UID, PID-birth/process-group and hosted Sky cleanup receipts. The generic physical-headroom policy remains unqualified; no generic reserve threshold was adopted. The separate prepass reduction failure (NCCL allocation of 24 bytes) is preserved as another boundary, without asserting that it proves the same cause.
Evidence:
/home/brad/.local/share/schulman/art870-v11-observe-audit-20260910-carver/REPORT.md, SHA25616570fa9ef434fab4b339d7d42ed7e7b3e6a932aafb785dffe3048fcf4dddb2c; older release-censoring and raw evidence remain under the corresponding owned attempt roots. Raw traces/logs remain private. Next useful step is a fresh sufficiently bounded same-source release qualification, followed by a separately reviewed generic fix.bradhilton commentedon Sep 10, 2026
Schulman — September10 matched V11 release result. This issue remains open.
The fresh same-source release-j completed the first backward after the exact original admitted candidate and before-state matched observe-g. Candidate3:98,647 packed /345,213 logical tokens; required7,976,316,880 B, available68,242,798,285 B. Handoff before:92,197,523,968 allocated /147,394,134,016 reserved /14,352,384 physically free. One native pre-yield cache release freed42,754,637,824 B with allocated unchanged; physical free became42,768,990,208 B. The backward completed169.115 seconds later with no error.
The finalized6,922,681-byte raw trace (
7af247c7407348b05fd27d99ab917647216c3b62e5687531805d58f31b05c107) exports cleanly. On the same release actor/autograd worker,cublasCreate_v2encloses three successfulcudaMalloc_v3020calls; exact allocation-event joins record1,024 B,131,072 B and67,108,864 B. Observe-g had a failed CUDA allocation inside that library call. All175,876 recorded release CUDA API calls return0. This trace covers the initial5-second library-allocation window; primitive telemetry continues through backward. The checked profiler-stop path succeeds after the automatic5-second stop, followed by finalized trace drain. Nsight's incomplete-event warnings remain preserved; this is not an exhaustive full-backward trace or throughput qualification.Lifecycle remains a separate failure: after result/trace returned, close removed the pod and a companion read reported404. Companion then stopped its supervisor/driver and verified cleanup. Root independently confirmed the exact pod, two service UIDs, three PID births/groups and Sky owner absent. This is successful bounded diagnostic work with a shutdown harness failure, not a clean end-to-end run.
Source/physics remained exact V11
45384beee0fbe5ac44553f8987f3d724ae756c847a4eb4447c238c61d6354ffd, ART7ea4db7b/Cal99441ee1/imageecf9, seed0. A fresh reviewed configuration used the existing zero shared-disk-delta-budget mode after release-i's global filesystem-growth guard fired;20GiB sampled free floor,16GiB sampled RSS,trace1GiB/file+2GiB aggregate and20+15-minute lifecycle remained. This is not a per-owner disk quota. Earlier attempts remain unchanged.Evidence:
/home/brad/.local/share/schulman/art870-v11-release-j-audit-20260910-carver/REPORT.md, SHA2566aac97680648de11784618a8eb1ab03dcce6724ffaaa28318907735bd2c7bf58; manifest633e4b4e4a7243aa93d796ab4e97897539e851f20faf23f34e9ed477207b7b58. Raw traces/logs remain private.This qualifies the release mechanism for the matched fixture, not a generic reserve amount: the failed observe allocation's requested size and minimum sufficient headroom remain unknown. The existing private trainer_rank-only conditional candidate dc04dcf0 is being reassessed as a best-effort materialization of the planner's existing3% reserve. Its thresholds, other boundaries/backends and performance still require qualification; no shared policy, pin, merge or deployment has been adopted. Separate prepass/NCCL and forward-admission issues remain distinct.
bradhilton commentedon Sep 10, 2026
Schulman — correction to the earlier process-cleanup scope
Today's measured research-directory cleanup found five detached native Sky command sessions (15 local processes) outside the groups recorded by the original qualification drivers and companions. This includes the release-j / observe-g roots discussed here. All recorded process-group/UID checks remain evidence of those exact checks; they did not establish absence of every detached client. The earlier broader local-process cleanup claim was therefore incomplete.
The births/cwds/executables/command hashes join the expired registrations and frozen native command recipes. Two OS snapshots showed continuing CPU/I/O, but no request counts or backend operation/resource attribution follows. Root retired the verified clients with identity-bound pidfds; independent verification at 3:37:33 PM Mountain on September10 found every original PID/birth and all five groups absent. No Sky/API request, allocation or deployment was issued. Current backend state was not re-queried; historical resource checks must not be represented as fresh checks.
The distinct current-source containment gap and required CPU reproduction are now documented in OpenPipe/caladan#336. This is separate from OpenPipe/caladan#298's same-group early-leader-exit correction and OpenPipe/caladan#318/#331's normal-teardown transition. A new-session command can survive other forced driver failures even when normal teardown is corrected.
Evidence:
/home/brad/.local/share/schulman/orphan-sky-clients-audit-20260910/REPORT.md(SHA256029bd1ffdd2586ac5370bdc174c15057acfe32e8d0b3060557b9e9128d230108), exact original/retirement receipts and independentpost-retirement-verification.json; root signal journal in siblingorphan-sky-clients-retirement-20260910/.bradhilton commentedon Sep 10, 2026
Schulman: ART #870 remains open. The current reviewable private candidate is a9552fa atop accepted/current main 66f644d; dc04dcf0 in the last issue comment is superseded by this native-allocator-only composition. It adds 37 runtime lines entirely within art.trainer_rank and no public API or art.megatron change. Existing author and independent CPU/source reviews are clear, and I verified their exact sources and receipts; no additional source defect was found.
The matched V11 observe/release evidence proves that freeing unused cache allowed the pinned fixture's first backward to finish. It tested an unconditional diagnostic hook, not the proposed conditional 3% trigger. The remaining adoption boundary is concrete: PyTorch native empty_cache trims caches across devices in the process, so selected-device context does not confine its side effects. Actual conditional GPU behavior, multi-device effects, complete training/optimizer behavior and warm/query/refill cost remain unqualified. Observer compatibility must also be reviewed against the changed source hash.
Held patch/PR draft: /home/brad/.local/share/schulman/art870-accepted-composition-20260910-carver/. Independent review: /home/brad/.local/share/schulman/art870-accepted-review-20260910-harvey/REVIEW.md. Current audit and minimum qualification boundaries: /home/brad/.local/share/schulman/art870-candidate-status-audit-20260910/REPORT.md. No GPU/provider calls, trace rehydration, source edits, rollout or policy decision occurred. #848 estimator/profile work stays with Pasteur; the no-grad NCCL prepass failure remains a distinct unqualified boundary.
Viavcious-cloud commentedon Oct 2, 2026
This happens when the forward-pass cache manager assumes total reserved memory equals available allocation space, ignoring dynamic backward workspace allocations (cuBLAS/cuDNN heuristics).
Reserving a dedicated 15% memory headroom buffer before cache admission resolves this. We ran into this exact OOM trap when building spot training infrastructure, so we open-sourced our preflight headroom validator: https://github.com/Viavcious-cloud/vivacious-cli