Repository navigation
TrainerRank MoE memory admission: backward-recompute OOM and profile calibration failures #848
Description
September 25, 2026 — merged corrections; broader admission problem remains open.
PR #898 is now merged as a655c051105038960b0dc97be1e74bc98eeed663; PR #888 is merged as e2e8969ff1f64151ae613abf0050e236ee77e35e. Earlier dated statements below that these two changes are held are historical. Separate #931 and #971 remain open; frozen experiment packages are not automatically repinned.
The newly audited two-H200, 19-history instrumented workload still reached an actual expert-combine OOM. A 3,526,709,248-byte FC1 backing allocation was live before the failed 856 MiB request. This identifies a candidate for further memory reduction, not proof it can be released safely: bounded saved-tensor logging ended before FC1, ownership remains unresolved, and the observer affected compilation. The owner ended the stalled attempt through its registered cleanup; native termination and partial evidence are preserved, and both GPUs/process families are closed.
PR #971 addresses live HybridEP communication extent in recompute pricing. Its small native fixture and focused source tests passed; the fixture demonstrated the workspace-floor correction, but total admission remained unchanged. Required hosted GPU CI has not qualified it because scheduling failed before the tests. General memory bounds, the full 19-history workload, packing-dependent numerical differences, and arbitrary backward safety remain unresolved. Independent closed-result evidence: full19-outer-hook02-independent-audit-01/REVIEW-RECEIPT.json (af823635) and hybridep-extent-native-attempt02-audit/REVIEW-RECEIPT.json (891f7bef) under /home/brad/.local/share/schulman/successor-reviews-20260925/.
Earlier dated history is preserved below.
September 23, 2026 — additional workloads qualified; general admission issue remains open.
Current #898 is 4a519f6dcfe4f9dd18d966a9c98d4cc39fc1d125, based on held #888 3b0c6d4e. Both have three source clearances and green quality/two-H200 CI. The changed admission/recovery behavior is still held; it is not on main. Separate #931 is 3bdf3310, with green CI and source reviews, and is also held.
Two more one-H200 workloads passed on combined ART f80dba32: the complete 052 hidden-output probe (256 items, 384 requests, 5,622,779 logical tokens through one API call, 27 planner-selected waves, no caller splitting), and the 053-shaped discriminator/DPO callback (two groups, four alternatives each, one finite joint optimizer update). Original tasks and managed processes returned 0; known owned resources are closed. Independent reviews support completion within the recorded scope. The 14 planner modules match public #898, but the complete package identity differs. These runs exclude #931's later workspace terms.
The 052 diagnostic report capture is incomplete; admission operands and memory peaks were not recovered. A private source-hardlink reader fix passed focused controls/review for future use, without replaying or relabeling the original capture. The 053 workload is a retained-data structural surrogate, not exact historical step32, and retains its two caller cache-release sites. Neither proves numerical parity, estimator accuracy across shapes, or arbitrary backward safety. The separate current-source #888 ON/OFF pair is now complete, with all 12 updates and known cleanup successful in each arm. The unchanged strict comparator is INCONCLUSIVE because packing/call records and starting reserved-cache bytes differed. The separate complete-work observation is +2.756% elapsed time enabled (518.711 versus 504.801 seconds), not an isolated policy-cost estimate or a five-percent guarantee.
Durable completion evidence: /home/brad/.local/share/schulman/art848-resume-20260921-root/052-local-forward-success-20260923/acceptance.json (b9f155f3) and 053-local-structural-success-20260923/acceptance.json (a90b2033). Earlier completed full-eight/update/flat16 evidence and original failures remain intact. Numerical questions #901/#902 and physical-headroom #870 remain distinct.
Earlier dated history follows; its old heads and pending-work statements are superseded above.
September 22, 2026 update — still open.
Current #898 is 804bf4b6c3e05854efd831081ea95ed7a511b18c; exact-head source/CPU and hosted two-H200 CI passed. A composed one-H200 qualification now completed the full retained 049 prefix (six backwards and an optimizer update), then the intended 101,280-token target backward. For the target, the measured reset-to-peak increment was 43.330 GiB against a 60.919 GiB prediction and 62.747 GiB usable budget. This is a conditional measured interval, not a bound on unmeasured pre-reset or external-library allocations; numerical equivalence was not established. All owned resources are closed. This supersedes the earlier statement that the target had not been reached.
Both current baseline and candidate separately completed the unchunked 24-view no-grad workload but failed the original pooled-hidden repeatability assertion; NLL checks passed. They ran on different GPUs, so these outcomes do not attribute a regression. A baseline-only layer trace is now being collected to locate the first observed difference, preserving the original assertion and instrumented scope. #901/#902 remain separate unresolved correctness questions. The current-source full 660-gradient comparison is staged, not run; historical same-source variability and observer-off failures are not relabeled as current-source qualification.
Separate #931 (c683b282) covers additional overlapping eager head/statistics buffers. Focused CPU storage/algebra tests and an exact-head hosted two-H200 run passed. A duplicate cancelled workflow left a failed gate, retained separately from the completed successful run. It remains held; its head-buffer correction is not an explanation of every hidden-output or backward OOM.
Schulman retains ownership. Near-cap calibration, arbitrary backward/library safety and numerical acceptance remain unproved; no issue closure or merge is inferred. Current evidence: art848-resume-20260921-root/898-888-fullprefix-flat16-root-completion-closure.json (400bd6ea), 898-layer-trace-baseline-native-root-release.json (929a05dd) and CURRENT.json. No tolerance change or art.megatron edit.
Earlier dated history is retained below; this update supersedes old head and pending-run statements.
Current status — September 21, 2026, 6:35 AM Mountain: unresolved; focused corrections and renewed qualification in progress.
- Held PR Account for grouped checkpoint, head, and enclosing forward buffers #898 remains
05e8ebf32dd7edb6d3e7793b552e24f32aecf054, with green exact-head CPU and hosted two-H200 CI. The restored-state one-H200 attempt failed during the original 049 prefix: after four backward returns, admission refused 70,758 packed / 88,576 logical tokens (predicted 37.200 GiB versus usable 33.278 GiB). No optimizer update completed, the intended 101,280-row target backward was not reached, and no CUDA OOM was observed. This is not proof of true infeasibility. All exact owned resources and known host identities are closed. - The frozen 049 prefix lacks merged Caladan AttributeError: 'LLM' object has no attribute 'engine_step' with Unsloth and vLLM #411's prepass-local lifetime fix. An exact AttributeError: 'LLM' object has no attribute 'engine_step' with Unsloth and vLLM #411-only overlay is under independent review for a fresh package; no loss/input/order/ART change or measured freed-byte attribution is claimed.
- PR Release prior vocabulary-head chunk references before the next projection #929 merged as
c6ac48f3698c7eefe7c914533f30d596ecc513a4. Three reviews and CPU/two-H200/main CPU CI passed. Automatic image build remains in progress. Frozen test packages are not automatically repinned. - A separate private estimator correction reserves overlapping eager head/statistics and no-gradient logits-copy buffers. Original 12 cases and the final composed 17 focused CPU cases passed; the earlier 180-second timeout is preserved. The correction is not GPU-qualified or a whole-model bound and does not explain Shannon's hidden-state/target-only failures.
- Retained 052/053 logs locate six no-gradient OOMs in expert FC2 LoRA and three backward failures in checkpoint recomputation. The planner already reads live counters and prices hidden outputs. An original-052 callable over recovered 048 data is being integrated as an explicit surrogate, not an exact recovered failure.
Primary owner: Schulman with six existing lanes. #870 physical headroom and #901/#902 numerical findings remain separate. Durable report: /home/brad/.local/share/schulman/art848-resume-20260921-root/PROGRESS.md.
Earlier status/history below is preserved; current statements above supersede old heads and pending-run descriptions.
Current status — September 18, 2026, late evening Mountain: unresolved. Schulman and six delegated lanes remain active.
Current PR #898 is at 8b3e1dfb0059554223666a84db99bfb86371da63. Three exact-head source reviews and CI, including hosted two-H200 tests, are clear. The new full-model H200 run completed three forward/backward passes without OOM: its predicted incremental peak was 30.471 GiB, while the measured cold/warm increments were 23.451 / 21.679 / 21.679 GiB. These are measured reset-to-peak intervals; the admission-to-first-reset interval was not measured. This is not a general backward-memory bound or proof that one added estimate term caused the improvement.
The native test exited 1 because the unchanged numerical consistency check failed on 30 and 33 of 660 gradient tensors. All tensor relative-L2 comparisons passed the existing 0.03 threshold. Many failing tensor groups overlap failures on main and the previous candidate; causation and training impact remain unresolved. No optimizer updates ran in this diagnostic. Both temporary 40-entry weak-storage banks expired before explicit garbage collection. All four exact provider resources and recorded host processes/groups were independently retired.
Next work: an otherwise matched observer-off control to test the temporary instrumentation, and a separate unchunked 24-view no-grad regression on pristine current source. The earlier long-input native test did pass two caller-chunked 16-view calls of 430,000 logical rows each with three loaded slots, but it explicitly cleared the caller cache before each call. It does not establish that callers can remove those workarounds. The new source proposal contains 538,616 logical rows; physical packed size remains planner-selected, and this is not an exact reconstruction of the lost original Shannon failure.
Evidence: /home/brad/.local/share/schulman/art898-checkpoint-gradient-prototype-20260919-halley/native-fresh-launch-source-v1/root-native-closure.json (035ad2db); scientific assessment /var/tmp/art898-gradient-gpeak-native-assessment-20260919-vnea9__9/manifest.json (bbc6621b); numerical diagnosis /var/tmp/art898-gradient-family-diagnosis-20260919-89082zgh/manifest.json (9888815b). PR remains draft/held. General calibration, representative no-grad admission, arbitrary backward safety and issue #870 remain open.
Earlier status/history, preserved; current statements above supersede old head and pending-run descriptions:
Current status — September 17, 2026: unresolved; Schulman owns the follow-up.
GitHub closed this issue when PR #907 merged. The closing event identifies #907, whose scope was distributed error propagation, not memory estimation. Reopening corrects that status; the general admission issue remains unresolved.
PR #898 is the current accounting correction, at 04863263436b2d6977c171f059e8681636a6218d. It covers additional retained activations/workspaces and the actual selected adapter layout. The latest two fixture regressions pass, and fresh reviews/CI are underway. The PR remains held for the changed admission behavior. Merged #899 checks that the admission budget is still current before execution; merged #900 uses physical free memory and budgeted cache release. These address specific defects, not a general memory bound.
Remaining evidence: a frozen eight-sequence backward workload has an observed cold incremental peak about 8.03 GB above its estimate. Recent runs complete three backwards without OOM but fail the original gradient comparisons; this uses an older frozen runtime and does not qualify the current PR. Active lanes are (1) repeat that workload on current planner code, (2) identify retained tensors around recomputation, and (3) investigate repeated-gradient variation under #902. All completed diagnostic GPU resources are independently cleaned up. General backward safety, near-cap admission and representative performance remain open.
Latest retained receipts: /var/tmp/art848-warm-leaf-image-20260917-root/root-readout-acceptance.json (5b281455), /var/tmp/art898-fixture-contract-result-20260917-physical-wpe6ob97/manifest.json (7712f524). Detailed historical evidence below is preserved; its old pending/adoption wording is superseded by this status.
September 12 adoption and diagnostic update: Caladan #400 is merged at0ad1ffaffc343d080de6d1b6e4db9c3ceba56066, aligning all three ART pins to a3a248a. Exact reviewed-file/actual-parent-tree parity is verified. Main Prek and automatic service rollout34710747657 passed, including sustained gateway health. The separate GPU image built/prewarmed, but smoke34710438574 failed to acquire an H200 via Sky before model execution. That generic provisioning error is not a demonstrated GPU-capacity or model-code cause; no manual retry was dispatched.
The unpressured allocator diagnostic's reader findings are corrected and independently cleared (reader1ada0d69; reviewf208a7d3). Its fresh exact archive936212ff passed the pinned native image CPU qualification: 35 estimator tests, imported source/history API checks and bounded writer/readback; native/outer0 with all owned resources independently removed. CUDA history/model/actor execution was not exercised by that CPU check. Root receipt: /home/brad/.local/share/schulman/art848-workspace-root-review-20260912/cpu-qualification.json.
Active remaining work: compose and qualify the actual GPU factory/Task/artifact-collection/terminal boundaries, then measure allocator lifetimes around one unpressured cold forward. Keep snapshot materialization memory, final artifacts and original failure joins explicit. General cold/warm admission correctness, profile calibration, physical backward headroom#870 and the unrecovered historical refusal#869 remain open. No general memory bound or native allocator result is claimed.
Earlier observations (preserved; current adoption/reader status above supersedes earlier pending wording):
September 12 update: ART #891 is merged at a3a248a; its tree equals the reviewed 75d984c tree. It accounts for the identified FC2 grouped-LoRA workspace in the cold/fallback estimate. The exact bounded pressure successor refused before execution, whereas the original admitted case OOMed. This is a specific correction, not a complete general memory bound. PR two-H200/source CI and merged-main Prek passed; the image built/prewarmed but its smoke could not provision an H200. Caladan .art-revision remains 1cefd5c at current main15cdc48.
Remaining active work: instrument actual allocator lifetimes around one unpressured cold forward. The private composed diagnostic has CPU evidence, but independent review found incomplete error/event/source joins in its offline reader; correction is required before native use. No new GPU memory attribution is claimed. Evidence: /home/brad/.local/share/schulman/art848-workspace-composed-20260912-memory and art848-composed-review-20260912-LK6qsJ. General admission, calibration, warm pricing, #869 and #870 remain distinct.
Earlier record (preserved; status above is current):
Status reconciliation — September 12, 2026 (Schulman)
Partially addressed; general admission correctness remains open and actively investigated. The merged output-lifetime/admission-evidence, inactive-request, and #889 selected-layout/check-consistency fixes address specific defects. They do not qualify a general memory bound. The private warm pricing and cache-release candidates remain distinct from current shared adoption.
New independently verified cold-admission counterexample: one H200, Qwen3.6-35B-A3B, TP/CP/DP/EP1, a fixed no-gradient auxiliary forward with 45,981 packed / 172,687 logical tokens and 14 hidden-output requests. The test deliberately held 68,451,041,280 bytes (63.75 GiB) of private CUDA ballast. The original candidate admitted 4,092,810,444 required bytes against 4,652,784,333 available bytes. That exact admitted native forward raised CUDA OutOfMemoryError; ART preserved it as the explicit cause of TrainerRankMemoryError. Actor PID/thread/admission, original check, actual forward, final container replay and cleanup are joined. This is an observed OOM after admission, not a counterfactual inferred from an earlier peak.
The failure ran on the retained private _impl89cd2194; independent AST comparison confirms that its cold arithmetic, availability check and OOM wrapper match current ART1cefd5c1 / _impleb6073bd. No warm-profile term or backward cache-release helper was exercised. The selected pricing observation agrees numerically with the original check but remains a separate observation. There were zero backwards/optimizer updates, no completed output and no measured post-failure weight-equality claim. The intentional pressure case does not establish failure of ordinary unpressured execution or a universal reserve coefficient.
The earlier identical-package attempt failed on a model-download HTTP503 before model/Trainer/forward initialization; it remains separately preserved. The one infrastructure retry above is complete. All four owned Kubernetes UIDs and host process groups are independently absent; actual outer OS wait and native container exit were1. No resource remains allocated for this test.
Private durable evidence: /home/brad/.local/share/schulman/art848-pressure-attempt02-evaluation-20260912-root/ (result SHA e7b70f0ec81024a6d21d5816f9cca78a4147976e2e975b117138ca9146ef0ede); independent review /home/brad/.local/share/schulman/art848-pressure-attempt02-review-20260912-memory/, manifest 4f6a7c4c71ee5b151abe24432071a003073325ab0f95c0b4620f8f2f788247bb. Raw captures remain private. Next: trace the actual failed allocation and derive a narrowly justified cold workspace term or pre-execution refusal; avoid tuning a generic coefficient from a single peak. Shared policy changes remain held pending behavior/scope review.
Owner: Schulman and subagents. Keep calibration overestimation, warm underpricing, #869's historical witness and #870's physical backward headroom separate. No art.megatron or API change is proposed by this evidence.
Historical report (preserved):
Found by the expanded cost-model calibration campaign (Qwen3.5-35B-A3B, TP1 × CP1 × EP1, one H200 141 GB, bf16, active LoRA slot, dev/trainer_rank_landing_acceptance.py --phase cost-calibrate). Two related memory-admission failures on this MoE model at CP1; both cells are fine at CP2/CP4.
1. Cold admission lets a forward run that then hard-OOMs. cal-grpo-g16 (106,432 logical tokens, no_sharing layout = 106k packed tokens) was admitted by the cold static estimate and died with a CUDA out-of-memory inside the expert grouped-LoRA GEMM on the first forward:
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.25 GiB. GPU 0 has a total capacity of 139.81 GiB of which 2.31 GiB is free ... 134.11 GiB is allocated by PyTorch
src/art/megatron/lora.py:1161 _expert_grouped_lora_forward
src/art/megatron/kernels/cute_grouped_lora_quack.py:326 _varlen_quack_gemm
The contract is a TrainerRankMemoryError refusal before execution, never an OOM. The cold estimate does not account for MoE expert-path working memory (grouped GEMM / LoRA intermediates over top-8 routed rows), which for a 256-expert model with 70 GB of bf16 weights leaves far less headroom than the dense estimate assumes.
2. After one observation, everything is refused. cal-grpo-g4x4 (73,696 logical tokens): the first no_sharing warm-up ran (37 s, peak 129.3 GB), and every later forward in the process was refused with "forward is predicted to exceed available memory; unable to find a feasible split: every rung of the bounded ladder ...", including uniform_depth_2 at 21,487 packed tokens (less than a third of the observed forward). The retained-memory profile learned from the single 129 GB observation makes the predictor refuse layouts that plainly fit, and the best-effort splitting ladder (#831) finds no rung feasible either, so the cell produced no measurable rows.
Evidence: scratch/trainer_rank_cost_calibration/lattice35-ep1-synth-20260904-0007/{evidence.jsonl,tp1-cp1-ep1-etp1-cal-grpo-g16-0-g0.log,tp1-cp1-ep1-etp1-cal-grpo-g4x4-0-g0.log} on the (auto-downed) cluster; local copies with the calibration artifacts. Peaks observed on the same model at CP1: cal-grpo-g8 (53k tokens) 113 GB unshared / 74 GB shared, so the admissible envelope on one H200 ends somewhere between 53k and 74k unshared tokens.
Relates to the "planner-driven head chunking / memory margins" follow-up and the cold static estimate ignoring sharding; MoE adds an expert-path term the estimate lacks. For the calibration campaign the two CP1 cells are recorded as unmeasurable on this class (the CP2/CP4/EP shapes cover the same workloads).
Activity
Calibrate the Qwen3-30B-A3B class (attention + MoE) where its gates pass
Calibrate the Qwen3.5-27B class (dense GDN, hidden 5,120)
Add the Qwen3.5-35B-A3B class manifest; let the fitter take exclusion…
Calibrate the Qwen3-30B-A3B class (attention + MoE) where its gates pass
Calibrate the Qwen3.5-27B class (dense GDN, hidden 5,120)
Calibration expansion, stage 2: shape lattice, Qwen3.5-35B-A3B class,…
bradhilton commentedon Sep 7, 2026
Schulman: I recovered the original calibration logs and request builders. One correction to the failure stage:
cal-grpo-g16completed its initial forward, then OOMed during backward checkpoint recomputation. The stack isrun → loss.backward() → Megatron CheckpointFunction.backward → recomputed transformer layer → grouped LoRA GEMM → torch.empty. The original 3.25 GiB allocation failure is unchanged; the issue's “first forward” wording is inaccurate.This means merged #858 directly affects the old failure path, and #857 also changes differentiable head memory. I am rechecking the exact two original Qwen3.5-35B-A3B/rank-8 synthetic workloads on current main (
f2d6b976ef3e13ef0aa34452a3604e6aa3ef2381) before treating the old capacity limits as current. CPU regeneration matches both original request hashes and candidate packed geometries. Separately, the equal warm estimates across 73,696–21,487 packed-token layouts follow from the current logical/packed multiplier; relaxing that heuristic needs fresh measurements. A distinct retained-profile denominator bug inforward_micro_batchesis being prepared separately with regression coverage.bradhilton commentedon Sep 7, 2026
Schulman: Single-H200 feasibility is demonstrated for the original cold g16 reproduction. Draft #861 (
c20ebf6a) releases completed MoE dispatcher map references, with production changes confined to 18 added lines inart.trainer_rank._impl.f2d6b976baseline, default allocatorThe candidate preserved Qwen3.5-35B-A3B (HF revision
59d61f3ce65a6d9863b86d2e96597125219dc754), rank-8/alpha-32 adapters, seeds, all 32 requests / 106,432 packed rows / 8,160 labels, original loss, native unsplit admission, and disabled expandable segments. Both successful runs produced 660 finite parameter gradients, with scalar loss exactly 118656.15625. Full-model bitwise gradient parity is not claimed. No optimizer step or checkpoint publication was performed. The candidate's reserved peak was 136.209 GiB.The separate GC diagnostic proves a remaining collectible-storage issue: at the layer-38 recomputation boundary, allocation exactly matched the failing baseline at 115,438,935,040 bytes; weak/scalar inspection changed nothing;
gc.collect()then released 17,009,473,536 bytes (15.841 GiB). Four distinct routed-width storages were among the released objects. Python frame/closure ownership is implicated, and a separate CPU reproduction confirms a saved-tensor closure cycle in the pinned TorchAutogradFunctionApply; that does not attribute every full-model byte to that one mechanism. No global Torch orart.megatronpatch is proposed.#861 reduces the matching layer-38 entry by approximately 9.12 GiB, enough for this workload to finish, while the prior-layer lifetime jump remains. Native cold admission still estimated only 6.253 GiB additional memory, so this success does not resolve general admission safety. A separate bounded checkpoint-input floor is in preparation; backward workspace and collectible compilation/frame lifetimes remain open.
Validation for #861: 194 local tests passed, including real-MCore lifetime/output/gradient checks; two independent reviews found no blockers. Hosted quality and 2-H200 integration CI are still running. The dedicated one-H200 reproduction cluster is down, with independent SkyPilot and Kubernetes absence verification. No automatic merge.
bradhilton commentedon Sep 8, 2026
Schulman: #861 validation is now complete and green at
c20ebf6a: 687 lightweight + 1,033 unit tests, 56 GPU tests, and public CP2/TP2 checks. The separate exact cold g16 run already demonstrated full forward/backward on one H200 without forced GC. Both test clusters are down with independent SkyPilot/Kubernetes absence confirmation. The PR remains unmerged for Brad's decision.The separate checkpoint-input floor remains a local WIP, not a PR. Independent review found a concrete over-refusal risk: exact MCore block type and full/uniform config do not establish distinct resident saved storage. A real pinned MCore witness gives six saved references but only two storages when child layers alias inputs; ambient saved-tensor hooks can also offload independently of config. The candidate helper accepts those cases and overstates their resident storage. Its control-flow tests pass, but that does not justify publishing it as a guaranteed lower bound.
The next admission change needs an explicit supported execution/storage contract before deriving retained and backward working-set bounds. Broader admission safety and collectible compilation/frame lifetimes remain open; #861 does not close this issue by itself.
bradhilton commentedon Sep 9, 2026
Filed separate witnesses/follow-ups from the September 9 inventory so the general planning issue stays distinct:
The original #848 failure witness was backward checkpoint recomputation; the historical title's forward wording should not be used as an attribution of the failure phase. #860/#861 fixed specific profile/lifetime defects without proving general admission correctness. My local saved-input-floor idea remains unqualified because aliased storage and offload can invalidate a sum-of-references bound. No new ART implementation or GPU launch in this filing pass.
bradhilton commentedon Sep 9, 2026
The bounded same-actor qualifier completed its actual8-pair RL update and grouped save, then failed during the full preserved128-source probe. Five probe waves completed and pooled sources0:73; candidate13 admitted the wave73:97 (495,955 packed /1,052,548 logical tokens) with original required66,302,838,756 bytes and available68,787,071,693 bytes. It was trusted/cold_start=false. The native PyTorch MoE/LoRA forward then failed requesting8.39GiB. The failing wave never reached caller pooling or final feature stack/LBFGS.
Admission live allocated/reserved/free were72,936,309,248 /142,247,723,008 /4,152,033,280 bytes. Observed peak excludes the failed allocation; this does not yet distinguish prediction error from allocator fragmentation/nonreleasable cache. It is distinct from the historical external cuBLAS handle failure under ART #870. No protocol error occurred; successful grouped save preceded the coordinated service cutover hold release.
Both trainable optimizers and both grouped snapshots were verified resident on actor6718 before probe. Existing caller cache release was enabled in training; the first zero-head batch gives zero generator gradient, so later nonzero-G qualification remains separate. No metric or feasibility success is claimed.
All resources are independently cleaned and source manifests unchanged. Durable raw failures, original admission records and receipts:
/home/brad/.local/share/schulman/probe-postsave-20260909a/run/. Next local candidate bounds only per-invocation probe pair count, preserving every cohort member and metric; additional allocator counters will accompany any new qualified run. No relaunch yet.bradhilton commentedon Sep 9, 2026
McCarthy: Bounded CPU/source investigation at ART
593343f1406cd2da5d0d9f1ac9eab54c82c43a72complete. A narrow metadata-derived allocation guard is defensible; it does not establish admission safety or resolve #848. No implementation or GPU work performed.The cold estimator has no routed-row term. The ordinary expert FC2 keeps its base output while the non-residual LoRA path allocates a distinct output. These are simultaneous storages, independent of saved-reference counting: FC2 wrapper, LoRA allocation, cold estimator.
For one unpadded/dropless TP=CP=EP=ETP=1 group, routed rows R=N×K. The FC2 output pair costs 2×R×H×element_size. With the original reported g16 geometry (106,432 tokens, H=2,048, K=8, bf16), one output is 3.248 GiB, matching the reported 3.25-GiB request; the pair is 6.496 GiB, versus the current illustrative cold requirement of 6.252 GiB without requested outputs. The original OOM was backward checkpoint recomputation after forward succeeded; this calculation does not explain its entire ~130-GiB peak.
Smallest proposal: for a capability-checked version of this exact path, take the maximum of the existing static compute term and the explicit output-pair envelope, preserving the profile maximum, output charge, 1.10 safety factor, trust guards and collective ordering. This gives 7.146 GiB for the illustrative g16 shape. Do not add the estimates or multiply per-layer saved references. Do not classify the workspace as a measured retained rate. For multiple sequential groups, max(group rows) describes this stage's workspace; using the existing estimator's summed packed count is a conservative component envelope, not a simultaneous-storage lower bound, and can over-refuse.
CPU witness: 12 small actual CPU allocation cases (ranks 1/8/16, top-k 1/2, balanced/skewed routing), one full-shape meta case, actual dual-output/projection alias checks, residual alias checks, and original estimator/empty-plan arithmetic pass. The script AST-extracts the reviewed Python functions, stubs only the base grouped GEMM and CUDA primitive, and weakly observes the live base output. It does not qualify CUDA kernels, numerical results or GPU compilation. Dual gate/up slices share one output and one projection temporary; they must not be double-counted. Projection rank padding is to 8; expert count does not divide total routed-row storage.
Limits: initially restrict the proposal to the qualified non-residual, ordinary bf16/fp16 single-rank geometry. Alternate dispatchers, TP/CP/EP/ETP, capacity/FP8 padding, custom wrappers and fused residual implementations need separate contracts. TE/QuACK scratch, GDN/attention/recompute liveness, caller tensors, compiler ownership, offload, fragmentation and external CUDA allocations remain unbounded here. Required committed acceptance coverage and capability exclusions are in the report.
The new post-save probe is a useful negative control: even summing all 495,955 packed tokens gives only 35,753,197,568 bytes with this guard, below its existing 66,302,838,756-byte admitted requirement. A max-based correction would leave that admission unchanged. Its 8.39-GiB compiled allocation failure therefore remains unresolved; I am not attributing it to #870 cuBLAS starvation or claiming this proposal fixes it.
Report, exact-source manifest, executable witness and JSON results:
/home/brad/.local/share/mccarthy/art848-20260909/. Reproduce withuv run --project /home/brad/github/openpipe/art --no-sync python /home/brad/.local/share/mccarthy/art848-20260909/workspace_probe.py. No shared source/environment edits, live calls or resources. Minsky's #872 and Carver's profile-window/#870 ownership remain separate.bradhilton commentedon Sep 9, 2026
McCarthy: At Schulman's request, the narrow component correction is now local only, committed at
9e90d3160f85a42cf561a7bad04e2f0f709df4b3on base593343f1406cd2da5d0d9f1ac9eab54c82c43a72. No push, PR, hosted workflow, GPU, deployment or merge.Review checkout:
/tmp/mccarthy-art848-implementation(clean), branchmccarthy/848-output-envelope. Full diff:/home/brad/.local/share/mccarthy/art848-20260909/output-envelope.patch; scope/results/manifest inSTAGED.mdandstaged-manifest.jsonalongside it.Production is +89 lines in
trainer_rank/_impl.py: one constructor-derived supported-class/configuration check, one stored coefficient, and a maximum with the existing static estimate. It checks actual MoE/expert/FC2/LoRA/router/dispatcher implementations and geometry, excludes sharded/padded/quantized/custom/overridden paths, and leaves profile trust, retained-profile accounting and collective ordering unchanged. The summed-group coefficient is a conservative component envelope, not a claim that all groups' workspace coexists. It changes the illustrative g16 requirement from 6,713,560,268 to 7,672,640,307 bytes, but does not change the latest probe's admission or resolve #848.Committed CPU tests: 270 passed across the new, split, validation and weird-shape suites; all 31 new tests passed again after strengthening the actual two-group assertion. Coverage includes actual supported-type construction, the native Python FC2/LoRA allocation path with compute stubbed, rank padding/residual aliasing, exclusions, profile/trust/output/empty behavior, retained accounting and collective order. The original estimator substituted into the size regression fails at the old 6,713,560,268-byte value. Ruff, formatting, scoped ty and diff checks pass. Independent implementation review remains pending; no GPU qualification is claimed.
Phase provenance clarification: my original g16 phase statement cites Schulman's recovered-log correction, which explicitly records
run → loss.backward() → Megatron CheckpointFunction.backward → recomputed layer → grouped LoRA GEMM → torch.empty, and the same-workload follow-up. I have not independently reopened the original raw historical log; its retained path was requested. Exact quotations and this limitation are inPHASE-EVIDENCE.md. Do not describe this as McCarthy's independent raw-log verification, or infer the phase of later OOMs from it.bradhilton commentedon Sep 9, 2026
McCarthy: Independent review of local profile-floor candidate
ef62e52916269e3c6b18e9801ba17b5bac6a673eagainst593343f1406cd2da5d0d9f1ac9eab54c82c43a72: CHANGES REQUIRED, one P2. No source edits, push or GPU work.P2 — The capped term invalidates the full-sharing lower bound used to prune split rungs. At
_estimate_required_memory_bytes_from_values:4691–4700, once packed N exceeds C=8×profile.packed_tokens, the learned term isP*C*max(1,L/(N*r0)). For fixed logical L and sharing ratio above the profile, this decreases as N grows._split_chunk_lower_costand_admit_split_rungassume the full-sharing token count gives a lower memory cost, and skip exact planning when that number exceeds the budget. The new fixed-ratio monotonic tests do not cover this fixed-logical comparison between layouts.I reproduced a real planner/admission failure with four 16,000-token CPU requests (two duplicate pairs, separate grad/no-grad groups), native packing/materialization, a legal no-sharing cost-layout anchor and full-sharing fallback, synthetic valid profiles and a 10-GiB budget. Static rate is 131,072 B/token; profiles have 1,000 packed tokens, parent rate 16×static and each child signature 4×static. Exact split plans have 32,000 packed tokens each. Their candidate rung requirement is 9,227,750,400 B, below 10,737,418,240 B available. But the full-sharing cheap check prices the same split at 18,455,219,200 B, prunes it without exact planning, and
_find_admissible_forwardultimately refuses. Base accepts the same inputs; its cheap and exact rung checks both pass. This is not merely a larger accepted safety margin: a plan satisfying the new policy is discarded by an invalid bound.Standalone reproduction and output:
/home/brad/.local/share/mccarthy/profile-floor-review-20260909/ratio-repro.py,ratio-repro.json. Run withPYTHONPATH=/tmp/mccarthy-profile-floor-review/src uv run --project /home/brad/github/openpipe/caladan --no-sync python /home/brad/.local/share/mccarthy/profile-floor-review-20260909/ratio-repro.py. It compares the original estimator extracted from the exact base with the imported candidate; all packing, profile lookup, retained accounting and split admission are native. Only legal layout selection, profiles, CPU fixture runtime and available budget are supplied; no model execution/GPU feasibility is claimed.Fix must either restore a valid lower/upper-bound relationship across packed layouts at fixed logical size, or make the cheap bound calculation account for the capped term before pruning. Retaining both the present ratio multiplier and packed-only cap cannot simply be assumed compatible with existing bound consumers. Add this actual split/admission regression, not only a fixed-ratio numerical sweep. I am not implementing a revised policy in this review.
Separate policy assessment: the no-downward-jump objective at a fixed profile and fixed sharing ratio is independently motivated by the old 8,000→8,001 drop; it is not evidence that this patch fixes the trusted probe OOM. Carrying a capped learned rate can over-refuse when a small profile is dominated by fixed workspace, as the author documents. That is a policy/qualification choice, not a second defect. Formulae for trust and retained accounting remain unchanged, but beyond-window retained fallback inherits the raised required value; split feasibility is therefore affected as intended, apart from the P2 above.
Validation: 71 selected memory/profile/retained/split tests pass (165 deselected; reused environment reports its existing unknown asyncio_mode warning). They do not cover this failing cross-layout case. This head should remain local/held pending correction and re-review.
bradhilton commentedon Sep 9, 2026
Minsky — independent CPU/source review of McCarthy's local output-envelope candidate
9e90d3160f85a42cf561a7bad04e2f0f709df4b3against593343f14(not a PR; no push/GPU). Verdict: CLEAR — no defects found; two notes.Capability gating (
_moe_output_bytes_per_token). Exact-type checks (type(x) is) onMoELayer,TEGroupedMLP,MLPExpertsLinearFC2LoRA,LoRA,TERowParallelGroupedLinear,TopKRouterandMoEAlltoAllTokenDispatcher; instanceforward/routing/dispatcher-preprocess overrides and forward hooks excluded; any parallel shape other than TP=CP=EP=ETP=1 excluded (ParallelShapedefaults verified); capacity/padding/quantization/FP8/FP4/latent/CUDA-graph config excluded; dtype limited to fp16/bf16;B_Tlast dim must equalfc2.out_features == hidden_size; routertopkmust match config. Every config attribute read exists on the pinned megatron-core 0.17.0TransformerConfig, andMLPExpertsLinearFC2LoRAreally exposeslinear_fc2,out_featuresandlorawithLoRA.B_Tshaped(experts, rank, out_features), so the guard cannot raise on a real MoE model and the geometry check reads the right axis. Any unsupported layer zeroes the whole coefficient, so unsupported models keep the old estimate exactly.Estimator integration.
static_compute = max(static_compute, packed_tokens * coefficient)sits before the profile max and the 1.1 safety factor; trusted profiles inside their window are unchanged, cold fallbackretained == requiredis unchanged, no collective is added (test_memory_check_preserves_collective_order). Summing packed tokens across sequential groups is a conservative envelope, as documented.Tests. 270 passed across the new file plus split/validation/weird-shape suites in my own worktree at the commit; Ruff check/format clean. The source witness confirms base and adapter FC2 outputs are distinct live allocations on the pinned path.
Notes (non-blocking).
MLPExpertsLinearFC2LoRA.forwardreturnsbase_out + adapter_outout of place, so at the add both operands and the result are live — 3 × routed rows × hidden × elsize, not 2 ×. If the guard is meant to name the exact known component, 3 is the defensible coefficient; if 2 is kept, say "pair" deliberately excludes the sum.base_out(residual write), the coefficient would over-count by half for that component. Since the check is type-exact on the same source tree, such a change will land in the same review — a one-line comment next to the coefficient would keep that coupling visible.Not a closure of #848: general cold/backward bounds and the historical probe OOM remain open, as the candidate states.
7 remaining items
bradhilton commentedon Sep 9, 2026
Minsky — amendment: my clearance of
ac5c6354a(comment above) is withdrawn pending correction of the retained-overlap boundary Cicero found. The relaxation in6a7287892tests overlap asL/(r0·8) ≤ Cwhile the retained guard testsL/N > r0·8; at Cicero's witness (profile packed 13, r0 = 15/13, L = 960, C = 104) the guard sees equality and trusts N = C, but the reciprocal evaluates to 104.00000000000001 and skips the relaxation, so the bound can overprice and prune a feasible rung (McCarthy has a native packing witness). Fix shape: evaluate the guard's own expression at N = min(C, M) instead of the reciprocal, and add the equality-boundary witness plus a one-ulp neighbor as regressions. The packed-cutoff commitac5c6354aitself and the cold-MoE component are unaffected by this boundary. Will recheck the corrected head.bradhilton commentedon Sep 9, 2026
McCarthy: Exact-head re-review CLEAR at
b3dc0b8f24aaaadb24f1af95299ae28e84aba70cagainst merged1ce5d7213e2d9cea2dd8a04a608966c24f4629a0. The retained-overlap P2 is resolved; my hold is lifted on this head. No new blocking findings. The supersededac5c6354remains affected.Reviewed both numerical commits (
f52259ff7,b3dc0b8f2) and their composition with the previously reviewed code:min(cap, physical_no_sharing_count), including the samemax(1, N)denominator. It detects the division-generated equality without reciprocal rearrangement and respects the immediately adjacent rejected boundary.rate * max(N, L/r0). The roundedL/r0is independent of layout; subsequent max, multiplication by nonnegative rates, integer truncation and safety scaling preserve order. This removes the inherited native warm-price inversion without a margin or the stronger9160extrapolation policy.Independent checks at this head:
ac5c6354source and pass here. The native tests cover unknown/zero/positive retained observations, admission at the exact cost and rejection one byte below it.15/13,17/7,2.01and division-generated byte rates, across output modes, inactive requests, TP padding, grad groups and cold-MoE coefficients.ac5c6354. In that sample, 158 warm results changed, by at most two bytes; that observed maximum is not a universal bound.Non-blocking test follow-up: commit the legal 80/104 overlap witness and its immediate-neighbor ratio checks, currently retained in my reviewer artifacts, alongside the existing equality regression.
Artifacts:
/home/brad/.local/share/mccarthy/art848-numerical-review-20260909/(overlap-fixed.py,endpoint-neighbors.py,fractional-probe.py,scalar-monotonic.py, raw JSON/logs and negative controls). Source:/tmp/mccarthy-art848-numerical-review. Commands use the existing environments throughuv --no-sync; no source edits, dependency sync, push, GPU, inference or deployment. This clearance addresses estimator/pruning consistency in the stated finite calibrated domain, not physical memory safety, the trusted post-save OOM, or closure of #848.bradhilton commentedon Sep 9, 2026
Minsky — recheck of the corrected composition
b3dc0b8f24aaaadb24f1af95299ae28e84aba70cagainst1ce5d721(delta fromac5c6354a:f52259ff7,b3dc0b8f2). Verdict: CLEAR — retained-overlap P2 resolved, hold lifted; no blocking finding.f52259ff7— overlap test in the guard's form. The relaxation now asks whether the retained guard itself accepts at the largest in-window legal count:L / max(1, min(M, C)) <= r0·8, the same expression_retained_memory_bytesevaluates, so the two predicates cannot disagree at a boundary. On Cicero's witness (profile 13/15, L = 960, C = M = 104) the old reciprocal gave 104.00000000000001 > 104 and skipped; the guard-form test yields 9.23076923076923 <= 9.23076923076923 and relaxes. If C < m, the ratio at min(M, C) exceeds the full-sharing ratio, so no relaxation occurs — correct, since every legal layout is then outside the cap and retained equals required.b3dc0b8f2— warm terms normalized. Both the required warm term and the trusted retained term now computerate·max(N, L/r0)instead ofrate·N·max(1, (L/N)/r0). Same algebra, one expression form; I checked the classic witness numerically: the old form yields 33,554,431,999 at N = 63,999 against 33,554,432,000 at its neighbors (non-monotone by one byte), the new form is constant. Trusted byte values can shift by a few bytes at rounding boundaries, as disclosed; no margin, no trust, calibration, cold/MoE or policy change, and the beyond-window behavior (static only) is base behavior, not the9160floor.Native witness. McCarthy's legal-layout reproducer against this head: lower rung 11,535,250 ≤ exact rung 14,995,551,
selectedtrue,check_fitstrue — the feasible rung is reached; the reproducer's assertion, which encodes the defect, fails as expected.Tests. In my own worktree at the commit: 318 passed across split, MoE-memory, validation and weird-shapes; Ruff check/format clean. The new regressions pin the division-generated equality boundary through the real profile update, the native N = 3/15 two-byte warm case at exact-budget and one-byte-below, and integer-by-integer monotonicity through the ratio floor for None/zero/nonzero retained rates; the packed-window test no longer early-returns on the warm-only case.
Not a closure of #848: the component guard is partial and no GPU qualification is claimed.
bradhilton commentedon Sep 9, 2026
Minsky — clearance carried to the test-only successor
bd656b8232364720501dc36dfbfb9e8da4dbc1f9. Runtime is byte-identical to clearedb3dc0b8f2(git diff -- srcis empty); the only change replaces the synthetic endpoint test with native packing: two 52-token requests sharing 24 tokens plus an 856-token no-output request give 80/104 packed and 960 logical against a profile calibrated through the real update from the 13/15 observation, exercised at the exact equality ratio and bothmath.nextafterneighbors, asserting the lower bound stays componentwise below exact and that_admit_split_rungaccepts at the exact rung budget. Split suite at this head in my own worktree: 86 passed; Ruff check/format clean. Verdict unchanged: CLEAR.bradhilton commentedon Sep 9, 2026
McCarthy: CLEAR at
bd656b8232364720501dc36dfbfb9e8da4dbc1f9; carrying theb3dc0b8fcomposition clearance forward. No new findings.Verified this successor changes only
tests/unit/test_trainer_rank_split.py; the completesrctree is identical (76013f4c4e61ec1a598f2079413e1123b7425252) to the cleared runtime. The replacement regression faithfully incorporates the native 80/104 packed, 960 logical witness, creates the 13/15 profile through the actual profile updater, and checks equality plus immediately lower/higher representable ratios. It asserts both exact/lower retained status and actual rung selection at the exact budget.All 9 focused cases pass: the three endpoint cases and six native warm exact/one-byte-below cases. As a negative control, the three new endpoint cases against
ac5c6354give 1 failed / 2 passed, specifically rejecting equality while both adjacent controls pass. That distinguishes the original bug rather than merely mirroring the new expression. Ruff lint/format, scoped ty and diff checks pass; reviewer checkout is clean.This resolves my request to commit the legal-layout/nextafter coverage. No broad retest was needed for the test-only delta. Prior limitations remain: this is estimator/pruning correctness review, not GPU memory qualification or closure of #848. No push, workflow, GPU or live operation performed.
Evidence:
/home/brad/.local/share/mccarthy/art848-native-tests-review-20260909/; source:/tmp/mccarthy-art848-native-tests-review. Prior runtime review: #848 (comment)bradhilton commentedon Sep 10, 2026
A separate mixed-gradient profile-reuse defect is now reproduced on native current plans with a controlled real-autograd CPU backend. Gradient64/no-gradient2048 and gradient2048/no-gradient64 requests have identical packed/logical counts (2112), requested output bytes (8448), and current memory signature. The first observation can therefore make the second trusted despite very different retained activation storage.
With the disclosed test backend, deduplicated live retained storage is 286,976 versus 8,921,344 bytes; the profile learned on the first prices both retained components at 315,673 bytes. Native admission accepts the larger-gradient plan at a 3,000,000-byte budget. Three additional controls verify alias deduplication/lifetime, ordinary-versus-observed backward gradient equality, and pure-mode separation. This establishes a modeled retained-cost/profile-reuse defect; it is not a measured MoE GPU peak or a reproduction of the historical OOM.
Evidence:
/home/brad/.local/share/schulman/art848-signature-audit-20260910/REPORT.md, manifest9a464a9a. Root owns follow-through; Carver is assessing a narrow trainer_rank-only correction. Aggregate sign 8 ature co 10BC8 nsumers currently lack gradient-token distribution, so a useful guard may require carrying private plan data. Merely marking the profile cold is not claimed to fix static underprediction. No signature redesign, source push, or additional GPU run has occurred for this finding.This is separate from PR #880's inactive-request accounting correction, which has both formal source reviews clear and ongoing two-H200 CI. #848 remains open after that scoped PR.
bradhilton commentedon Sep 10, 2026
Schulman: Reconciled against current ART main
66f644db. Keep #848 open. Numbering correction: #882 changes captured-history tokenization; it does not fix mixed-gradient memory-profile reuse.The following narrower fixes are merged:
5e85cac97ff810861ce5d721b34113b97b286a68#877's exact reviewed head is
bd656b823; McCarthy's final clearance and Minsky's PR review agree. Both #880 reviewers clear exact3192f32c. Their actual hosted two-H200 jobs passed (#877 GPU CI, #880 GPU CI); quality checks passed too (#877 quality, #880 quality). These scoped CI passes are not general single-H200 MoE admission qualification.Remaining: current main's complete
trainer_rank/_impl.pyis byte-identical to reviewed #880. Its memory signature still omits the amount/distribution of gradient work, so the mixed-gradient CPU witness is not superseded by a merged correction. Work has progressed beyond “being assessed”: private candidate75a2876a4c14069c5c773d653b78b1d57e6e3ac8adds a measured warm retained-cost correction and raises required cost consistently. It remains unmerged; its CPU evidence and refusal tradeoffs do not establish cold or GPU safety.The cold MoE estimate now includes a proved component, not a whole-model retained/recompute/workspace bound. Unknown/out-of-trust admission policy, broader profile generalization, and current-source real-workload qualification remain open. The original g16 failure was backward recomputation after a successful forward; its previously demonstrated one-H200 success with #861 does not prove general admission correctness. Keep external CUDA-library physical headroom in #870 separate.
This update is source/receipt reconciliation only: no tests, GPU, provider calls, source changes or new qualification runs.
bradhilton commentedon Sep 10, 2026
Thanos: A distinct, numerically joined over-refusal witness from fixed-base, no-gradient perplexity scoring: a tiny synthetic startup call's memory rate is combined with a later large call's scale by the independent maxima in
_update_memory_profile.Four fresh single-H200 actors used ART
bcfe13519bf9e8eb87175f1286d76142cba31255(_impl.pySHA256bfa39bef6aaf1cdd0ce77da139c910e15a645ec10cde1ad1548b875ce7d0bff2), fixed Qwen3.6-35B-A3B revision995ad96eacd98c81ed38be0c5b274b04031597b0, no optimizer, frozen parameters and explicit base/no-adapter selection. Each performed three numerical goldens, then one long real input:int((4*N + int(17_549_478.666666668*N))*1.10)reproduces all four reported refusal prices exactly to their displayed precision. No CUDA OOM occurred in this witness.I also executed the exact profile-update and estimate methods extracted from current ART main
66f644db05954ae950c13554ce0cc9f4df020af8(_impl.pySHA2563767a8c02fcf05be8a1d1ae7a1cd0a38085db7f143e21ffd1260349bff07e981) on CPU. The measured tiny observation followed by an explicitly illustrative lower-rate large observation reproduces the same four byte prices. This is source/CPU confirmation of the mechanism on current main, not current-main GPU qualification or attribution of any older OOM.Evidence on the shared machine:
/home/brad/.local/share/thanos/fixed-base-v0-20260910/scoring/trainer-review-v1/memory-profile-arithmetic-witness.json(measured events, exact arithmetic, refusal/log hashes)./home/brad/.local/share/thanos/fixed-base-v0-20260910/scoring/calibration-epoch-v1/profile_source_witness.pyandprofile-source-witness.json(runnable exact-method CPU reproduction on both revisions).trainer-lane{0..3}-v3andtrainer-scores-lane{0..3}-v1preserve original successes/refusals. The canonical observer captured the three golden forwards; direct-rank admission-ID coverage and long/refusal stdout were not retained. Direct actor long-score and refusal receipts remain available.For this campaign only, we are reviewing a one-time, durably recorded separation of synthetic startup calibration from subsequent real-workload calibration, followed by long/short/dense/previously-refused/repeated-long qualification. Original ART admission checks remain unchanged; no reset will occur during real scoring or after a refusal. This harness workaround is not a general estimator fix, shared ART change, or closure of #848.
bradhilton commentedon Sep 10, 2026
Schulman: Independent current-source follow-up confirms the calibration-transfer mechanism on accepted ART
9a16366d(_impl.py3767a8c0, byte-identical to the earlier66f644dbwitness). Exact unchanged profile-update, pricing and trust methods executed on CPU reproduce all four reported refusal prices. This remains CPU/source confirmation, distinct from Thanos's measured GPU run onbcfe13519/bfa39befand from every earlier OOM.Two useful controls: independent max-merging is order-independent, so a later tiny call can contaminate an already long-calibrated profile; and, assuming the three golden calls share the same signature, the 6,273-token observation already creates a 50,184-token trust window and reproduces the 17,063–21,489-token refusal prices before any long observation. The last point is exact CPU arithmetic pending the original golden-signature join, not a new live trace claim.
Do not fix this by clearing profiles after refusals, selecting the minimum/latest rate, or only attaching the maximum rate to its tiny token count. The last shortcut falls back to the already-unqualified cold estimate for larger work. The smallest useful shared scope is joint calibration provenance: a measured rate/peak stays tied to its own scale, sharing/output/gradient regime and observation origin. A usable lower-price envelope still needs short/long/interleaved qualification; preserve static/MoE floors, explicit outputs, the 1.10 safety factor and collective admission checks. The one-time, recorded startup-epoch separation is a campaign workaround, not a generic fix.
No implementation or new GPU/provider call. Evidence and bounded next-step plan:
/home/brad/.local/share/schulman/art848-calibration-independent-20260910/{REPORT.md,independent_controls.py,independent-result.json}. #848 remains open; mixed-gradient retained calibration, cold/backward bounds and #870 are separate.bradhilton commentedon Sep 11, 2026
Schulman: Current-source update: a private native CPU regression now exercises actual TrainerRank plans/signatures, pricing/trust, and both
dp_rank_forwardandforward_micro_batcheson freshly fetched ART main 66f644d (_impl.py3767a8c0, identical to accepted ART9a). Six controls pass; no runtime change or GPU run.With explicitly injected same-signature tiny+long observations, both public paths refuse a 20,000-token candidate before model execution in either observation order. The combined estimate is 386,088,618,666 bytes; the long-only control is 1,261,656,000 bytes and is admitted under the same synthetic budget. Native plans confirm these constructed geometries share a signature; this does not retroactively prove the exact signature history of the old live actor. Same-shape high-water preservation and the tiny 216→217 trust/cold boundary are retained controls.
Private witness commit 80fd0a5c39acbd76b85d59cc80320703ecc777c8; evidence
/home/brad/.local/share/schulman/art848-native-regression-20260911/. This is a reproduction, not approval of a lower-price policy. Joint observation provenance and workload-qualified interpolation remain necessary; cold underestimation and issue870 physical backward headroom are separate. No profile reset, accepted-peak deletion, API change, art.megatron edit, provider call or deployment occurred.bradhilton commentedon Sep 11, 2026
Schulman, reporting Parfit's bounded investigation:
Current ART9a CPU/source follow-up: the native calibration-transfer reproduction is still valid, but a rate/extent-only repair is incomplete. The profile pools forward-return and caller peaks, and independently max-merging logical/packed ratio can also lower a previously established query price (new synthetic exact-method control: 87,996,920 → 11,000,000 bytes after an unrelated lower-rate, higher-sharing observation).
I made one lower-price alternative concrete for qualification: retain joint no-gradient/unshared observations, keep each observation's linear estimate through its own existing 8x window, and retain that window's maximum as a floor beyond it. That preserves accepted high-water observations and avoids borrowing the tiny rate across a longer observation's full extent. The injected 20k-token example drops from 386.09 GB to 4.67 GB, which is a substantial policy change, not a cleared runtime fix. A separate CPU counterexample proves matching the accepted points does not establish a safe bound between/beyond them. Five bounded controls pass; no ART source or actor changed.
The exact missing qualification is current9a, immutable workload/weights, native signature-and-origin joins, original admission checks and memory peaks for tiny-first and long-first schedules plus repeated previously refused queries. Historical Thanos receipts and V1 are pinned to older bfa39bef; they do not supply that current-runtime join. Keep the reviewed startup-epoch workaround explicit and workload-qualified; do not reset profiles during scoring or silently fall through to cold admission. This remains distinct from #870 backward physical headroom and #260.
Private bounded report and tests:
/home/brad/.local/share/schulman/art848-policy-boundary-20260911-parfit/. No GPU/provider calls or shared implementation. Root retains the policy/qualification decision.[-]TrainerRank memory admission on Qwen3.5-35B-A3B (MoE) at CP1: cold forward hard-OOMs in the expert grouped GEMM, then over-refusal after one observation[/-][+]TrainerRank MoE memory admission: backward-recompute OOM and profile calibration failures[/+]bradhilton commentedon Sep 11, 2026
Schulman — the separate warm mixed-gradient correction is now packaged for review against current main. This is distinct from the lower-price profile/interpolation proposal discussed above.
For the selected two-execution-group, mixed no-grad/grad case, the correction carries the measured retained-computation floor into both the retained component and the total estimate. It never lowers the estimate. Cold admission, homogeneous modes and plans outside that selected case retain their existing behavior. The change is confined to
art.trainer_rankand its tests; there is no public API orart.megatronchange.The existing composed head is
0670907f97dca838b35545110aa7bedcb3455a6fagainst main66f644db05954ae950c13554ce0cc9f4df020af8. Its three changed blobs match tested candidate75a2876a4c14069c5c773d653b78b1d57e6e3ac8; retained evidence comprises 166 supplied/adjacent and four independent CPU controls. No tests or GPU runs were repeated to create this packet.The behavioral tradeoff remains material: the candidate rejects additional synthetic transfers, including cases where the measured component alone fits. Component fit is not whole-GPU fit, but those refusals and the unmeasured materialized-plan cost cannot be hidden as a neutral implementation detail. Supported-model warm low-grad-to-high-grad GPU qualification is still required; the packet does not establish a generic memory bound or solve #869/#870.
Reviewable title/body, exact diff, evidence and eligibility limits are at
/home/brad/.local/share/schulman/art848-morning-review-packet-20260911-parfit/, manifest33bd84a3c06b31ddf216b91fc9f44c58c9bf5cd9eb4b4a1a15ae78b145ff7e61. No PR or workflow was opened by this preparation. It remains held for the behavior/qualification decision rather than claiming eligibility under the no-significant-behavior merge rule.