TrainerRank CP2/EP1 normal admission reaches CUDA OOM in compiled MoE LoRA forward

A fixed, offline synthetic diagnostic on two H200s, CP2/EP1 reached a CUDA OOM during the first large grouped forward after the planner selected admitted. Its selected required estimate was below the available budget. Oversized admission was enabled, but this event did not select admitted_oversized.

This is distinct from #948, which reported an explicit oversized override on CP1. Both traverse MoE/LoRA forward code; neither report establishes an identical cause. This new diagnostic preserves the original nested callback grouping, uses synthetic inputs/adapters, and does not reproduce the historical production batch or establish a regression on current main.

Actual failure, September 24, 2026, 7:01:25 AM MDT (13:01:25 UTC): rank 1, event 51a626adb7fe493c9726a222d92d9cf2, forward phase. The inner exception is torch.cuda.OutOfMemoryError, wrapped by TrainerRankPartialExecutionError. The outer harness's shallow cuda_oom:false classification describes the wrapper and is superseded by the retained inner exception/planner report.

The allocator message reports an attempted 6.68 GiB allocation with 6.07 GiB free, device capacity 139.81 GiB, process use including non-PyTorch memory 133.61 GiB, PyTorch allocated 113.02 GiB, and reserved but unallocated 14.74 GiB. These rounded message values do not prove fragmentation. The separately retained allocator counters are explicitly sampled post-unwind.

Selected admission / report field	Bytes
Predicted peak above observation baseline	44,681,050,584
Admission required, safety factor 1.1	49,149,155,642
Available admission budget	66,481,491,149
Physical free at selected admission	70,985,121,792
PyTorch allocated at selected admission	70,191,249,408
PyTorch reserved at selected admission	72,706,162,688
Device capacity	150,121,021,440
OOM partial peak above observation baseline	54,895,467,520
Observation baseline	70,191,232,512

The partial peak is a lower bound from the interrupted observation, not a completed peak. It already exceeds the predicted peak and the admission estimate. observed_peak_bytes and error_pct are null; this report does not manufacture a completed underestimate percentage. Selected ordinal 5 carries the required estimate from ordinal 4; allocator is native, scope world, reserve 3%, no test memory cap. Initial required bytes were 1,944,298,502,665, followed by native replanning to the selected estimate.

Failed execution plan: 10 requests, 779,067 logical and packed tokens, two subforwards, with packed-token counts 400,511 and 378,556 and five GDN segments each. Gradients enabled, logits/hidden-state outputs disabled. Planner topology (DP, TP, CP, PP)=(1,1,2,1), coefficients [2,"gdn-moe-h2048-h200-bf16"]. No full-size forward wave completed and no full-size optimizer step occurred.

Subforward	Request lengths in retained plan-group order	Logical/packed tokens
0, synthetic policy	94017, 89049, 80763, 72480, 64202	400511
1, synthetic adversary	93189, 84906, 76618, 68338, 55505	378556

The retained stack ends in megatron.core.transformer.moe.experts.forward, ART expert FC2 LoRA, and an Inductor-generated add. The actual generated file was preserved before cache retirement. Its failing line 98 is:

s38 = arg1_1.size()[0]
assert_size_stride(arg1_1, (s38, 2048), (2048, 1))
assert_size_stride(arg2_1, (s38, 2048), (2048, 1))
buf0 = empty_strided_cuda((s38, 2048), (2048, 1), torch.bfloat16)

The runtime value of s38 was not retained, so the symbolic shape plus rounded 6.68 GiB message is not an exact element-count measurement.

Reproduction configuration and ordering:

Model Qwen/Qwen3.6-35B-A3B, revision 995ad96eacd98c81ed38be0c5b274b04031597b0, BF16, LoRA rank 1 / alpha 32. DP1/TP1/CP2/PP1/EP1/ETP1 verified on both ranks; full uniform recomputation, one layer, compiled transformer layers. Each CP rank receives the identical complete input tree, not half of the 19 histories.
Python 3.12.13; Torch 2.11.0+cu128; CUDA 12.8; Megatron Core 0.17.0; Megatron Bridge 0.5.0+e1a207ac; Transformer Engine 2.11.0 with the frozen empty-group patch; Transformers 5.12.1. Two local H200s, 16 CPU affinity, OMP/MKL/OpenBLAS thread counts 1. This is an isolated exact runtime, not shared-environment package substitution.
ART base 8ff205e543473d96557fe4b0ffbeaea19d13b420 with existing feature/instrumentation backports; Caladan base 257be1824d2dd7495e6e3509fbcad9af9ee74e9a. Executed trainer_rank/_impl.py SHA-256 6c0e9e68d32eb0067fa5206f0ff419c2840fd60922f0e90941c07b4486b1f313. This is not a clean-base repro.
Both actual ranks assert ART_TRAINER_RANK_ALLOW_OVERSIZED_BATCHES=1 and ART_TRAINER_RANK_PLANNER_MISS_THRESHOLD_PCT=10. The estimator/admission implementation was unchanged by this diagnostic; bounded report retention was configured separately.
Use native collective adapter export and checkpoint loading to put identical synthetic weights into policy/adversary slots. Nonzero LoRA initialization, native runtime seed 593015. Rank 0's adapter is authoritative; synchronize before export and agree file completion before either rank loads. Reset both slots to the same adapter with absent optimizer state before each phase.
First execute the original grouped callback on lengths [64,56,72,64,64,56,72,64], two 8000 groups of four. This tiny phase passed forward/backward, one optimizer step, counters and CP replica checks. Reload both initial slots, restore baseline RNG, then run the large phase below; keep the warmed runtime/planner state.

lengths = [94017,93189,89049,84906,80763,76618,72480,68338,64202,
           55505,51373,47238,43111,38974,32251,28111,23973,16565,12430]
g = torch.Generator(device="cpu").manual_seed(593016)
tokens = [torch.randint(16, vocab_size, (n + 1,), generator=g,
                        dtype=torch.int64) for n in lengths]
# Alternate policy/adversary slots by history index. Use the unchanged nested
# callback on groups of 10 and 9 whole histories, not singleton backwards.
# Sample mask: arange(n+1) % 7 == 2; first position false; logprobs -10.
# Synthetic group rewards +1/-1. Token values are independent uniform IDs.

These histories total 1,073,093 tokens. They deliberately lack production prefix sharing, routing values, trajectories and actual batch membership; the 10+9 grouping is synthetic. Original callback source SHA-256 55a7eea915d3709820b9a3cbd90d98468701edbbfe785d1552c9d6625f37dfab; exact two-rank harness SHA-256 0df7393c12ef8fa67665cbb3652d2fb1eabb56c7010bed41c028404a311110af. The complete harness, synthetic fixture and all five planner reports are retained. The metadata report alone has replay_complete=false (device_or_modified_input, layout_inputs_unavailable); this issue does not claim the JSON alone is an executable GPU reproducer.

Source questions: the frozen _moe_output_bytes_per_token returns zero outside TP1/CP1, and this CP2 report records zero. Full recomputation uses the generic static compute estimate; the split-memory floor is zero and is only learned after successful completion. These are concrete places to examine CP2 compiled MoE working-set coverage and smaller native subforward plans while preserving the callback grouping. They are not proven fixes, and the allocator's fragmentation suggestion has not been tested.

Expected behavior for normal fit admission is either successful execution or a conservative refusal/replan before this allocation. Actual outcome is terminal CUDA OOM. This diagnostic provides no full19 memory-fit or qualification credit and no cross-topology numerical-equivalence claim. No retry or proposed fix was run. All owned ranks, processes and guardian were verified gone, assigned GPUs empty, and private runtime/compile bulk retired; evidence remains outside cleanup targets.

Retained artifact SHA-256 values: original planner report b24df6f4d604e309247cad106557ac893ac5e9441ac1ca8e283d423cd214694f; sanitized planner projection b70ac2e7cd761ed4d3a69d18974eb4d277d82ea6f05807f07cdf97ae549bbf2f; failed generated code fb9c32f5ab99e0d07b892b29a53b9273a141b1622935b630cc123f0549a593fe. No private text, credentials or checkpoint weights are included in this issue.

A separate CP2/EP2 cold full19 diagnostic also OOMed on two H200s. This preserves the original nested backward grouping and starts a fresh TrainerRank directly on the large synthetic phase, with no tiny calibration and no profile clearing. It is additional synthetic reproduction evidence; it does not establish a production replay, cross-topology numerical equivalence or qualification.

Both ranks successfully constructed the runtime, loaded the retained complete synthetic adapter through the native loader, and passed export/reload and canonical ownership checks: expert keys have one owner and a complete global union, while dense replicas agree. Actual topology was world2, DP1, TP1, CP2, EP2, PP1, ETP1. The pre-full profile list was empty on both ranks. The first grouped forward failed before any completed wave, backward or optimizer step; rank1 was then stopped during peer-failure cleanup.

The planner admitted normally, with oversized admission enabled but no oversized override used:

Field	Bytes
Predicted incremental peak	44,681,050,584
Required with safety factor1.1	49,149,155,642
World-reduced available	105,054,407,885
Baseline allocated	37,455,549,952
Partial incremental peak lower bound	83,240,558,080

The cold plan had10 requests in two slots,779,067 logical/packed tokens, and one unsplit subforward. Its profile was null; moe_output_bytes_per_token=0, full recompute was active, and the miss threshold was10%. observed_peak_bytes and error_pct remain null because the forward did not finish. The partial peak is not a completed measurement.

Rank0’s CUDA error at 8:07:32 AM Mountain (14:07:32 UTC) reported a6.54GiB allocation request with6.24GiB free on a139.81GiB H200,106.35GiB allocated by PyTorch and19.05GiB reserved but unallocated. The latter is allocator context, not proof that fragmentation caused the failure. Post-unwind allocator counters are retained separately from the failure message and partial peak.

The bounded primitive trace reaches MoE experts → art.megatron.lora.forward → generated Runner.call, line98:

buf0 = empty_strided_cuda((s38, 2048), (2048, 1), torch.bfloat16)
# aten.add: base expert output + LoRA output

The actual failing s38 was not retained. The generated file’s benchmark get_args() exemplar is not evidence of that runtime value.

Reproduction bindings:

Qwen/Qwen3.6-35B-A3B revision995ad96eacd98c81ed38be0c5b274b04031597b0; BF16; two H200s,150,121,021,440 bytes each.
Frozen ART8ff205e543473d96557fe4b0ffbeaea19d13b420 plus the retained diagnostic/instrumentation source; TrainerRank _impl.py SHA6c0e9e68d32eb0067fa5206f0ff419c2840fd60922f0e90941c07b4486b1f313. Caladan257be1824d2dd7495e6e3509fbcad9af9ee74e9a original nested callback unchanged.
Python3.12.13, Torch2.11.0+cu128, Megatron Core0.17.0, Bridge0.5.0+e1a207ac, patched Transformer Engine2.11.0, HybridEP1.2.1.post1+art.1c10720b3830d561 (SM9.0, single-node, NIXL disabled).16-CPU affinity, single-thread numeric libraries. Required launcher flags were ART_TRAINER_RANK_ALLOW_OVERSIZED_BATCHES=1 and ART_TRAINER_RANK_PLANNER_MISS_THRESHOLD_PCT=10.
Full19 lengths: [94017,93189,89049,84906,80763,76618,72480,68338,64202,55505,51373,47238,43111,38974,32251,28111,23973,16565,12430]; original nested groups10+9; the first group is the failed call. Public synthetic tokens use CPU generator seed593016, alternating policy/adversary histories, the retained complete synthetic adapter, and unchanged masks/rewards/logprobs from the pinned harness.
Cold harness SHA58e9b18feb7de8e0e5f35231fc678a0ee52b2566ab4a74726cca1b1eacdb4623; ownership helper740a1123dbb226d2b583822b9843be45a6b026118924ab16b69140f0421b6f26; adapter safetensors679c53a38b19fa44b1ced3863e5e6b7c362b14ec86b28acb9cc50adfaf075ce5. Exact command, source/setup seals, all-rank journals and primitive traces are retained with execution328849ebde454566b0d8fa4c5e56bbac.

Planner reportb76f8cd250614cc0aeeb6c9b5b0c6a01, SHA d39edcf3779e928ed1af00a653049f560760572c674f5baf7fdb8cb162ab4651, is preserved alongside a sanitized copy. It says replay_complete=false because device token contents/layout inputs are absent; the report alone does not provide a GPU replay. The pinned synthetic harness, adapter and runtime are needed. Generated failure source SHA4adc333318932944f15fea18f22b97cc4702d2c1454e74fec91e7749cdb9d0c7 is retained.

All six owned process identities (including the independent guardian) closed, and both GPU UUIDs were compute-empty at 8:08:52 AM Mountain (14:08:52 UTC). There was no automatic retry.

Allocator comparison clarification for the CP2/EP2 cold diagnostic: the standalone launcher sanitized its environment and did not set either allocator configuration variable. Frozen Caladan's Qwen trainer defaults specify PYTORCH_ALLOC_CONF=expandable_segments:True; frozen ART's Monarch worker bootstrap assigns the same value to both allocator aliases. The standalone torchrun harness does not traverse that worker bootstrap. Its recorded native allocator backend is insufficient to establish whether expandable segments were effective, and no effective-settings receipt was retained. Historical CP2 effective configuration therefore remains unknown; this diagnostic is not proof of allocator-setting parity with production.

The 19.05 GiB reserved-but-unallocated figure does not prove fragmentation caused the failure or that a reusable 6.54 GiB block existed. The retained partial incremental peak already exceeded the cold prediction, but it is a lower bound rather than a completed peak. The next original-topology diagnostic is being prepared to match both frozen allocator aliases and record only the effective scalar settings before model construction. No retry result, allocator fix, successful fit or production reproduction is claimed.

Exact installed Torch git70d99e998b4955e0049d13a98d77ae1b14db1f45 accepts programmatic setting changes and checks the legacy CUDA alias before the unified alias, so environment absence alone is not a conclusive effective-state measurement. Setting both aliases identically avoids that precedence ambiguity. Matching Torch allocator parser.

A four-H200 CP4/EP4 diagnostic passed at 8:36:04 AM Mountain on September 24. All four ranks completed the original nested 10+9 grouping, both forward/backward waves, one native optimizer call, and synthetic parameter counters from 0 to 1. The full 19-request synthetic geometry and original callback were preserved; there was no tiny calibration or profile clearing.

Two things changed relative to the failed CP2/EP2 diagnostic: CP/EP degree increased to 4, and the launcher matched the frozen production allocator default, expandable_segments:True. Before NCCL/model construction, all four ranks recorded the native allocator backend, effective expandable segments enabled, graph-capture-record-stream reuse disabled, and both environment aliases matching the parsed default. Only allowlisted scalar settings were retained. This result cannot isolate the topology effect from the allocator effect or establish fragmentation as the earlier OOM cause. The previous direct EP2 launcher omitted both allocator variables, and its effective parsed setting was not recorded.

The planner still substantially underestimated the first wave. With oversized admission enabled and miss reporting at 10%, eight completed miss reports were emitted. Every plan was admitted normally; there was no oversized override or OOM.

Rank	Predicted incremental peak	Observed incremental peak	Underestimate
0	44.681 GB	89.140 GB	99.50%
1	44.681 GB	84.804 GB	89.80%
2	44.681 GB	100.889 GB	125.80%
3	44.681 GB	93.568 GB	109.41%

GB means 10^9 bytes. The first wave contained 10 requests, two slots and 779,067 packed tokens in one unsplit subforward, with no prior profile. Admission required 49,149,155,642 bytes against 120,357,326,029 world-reduced available bytes. The observations cover forward_and_caller, including backward, and are completed incremental measurements; they are distinct from absolute forward-only peaks. The second nine-request wave also completed normally; its forecasts overestimated the observed 6.087–8.051 GB increments.

Actual topology was world=4, DP=1, TP=1, CP=4, EP=4, PP=1, ETP=1. Model 995ad96e, Torch 2.11.0+cu128, patched Transformer Engine 2.11.0 and HybridEP 1.2.1.post1+art.1c10720b3830d561 matched the prior pinned runtime. All four ranks passed synthetic adapter load/export/reload and global expert ownership checks. Each rank recorded 160 backward layer-recompute calls and 1,320 native parameter-counter transitions.

This establishes completion of one grouped update for the declared synthetic geometry under the combined CP4/EP4 and allocator configuration. It does not establish cross-topology numerical equivalence, production checkpoint-225 portability, 300 optimizer steps or sustained GPU qualification. Production tokens and routing may require different memory.

Reproduction evidence is preserved under execution 685121a27aef4287860feb38a4616aa9, source seal b415e440890e76a0d574213c8cad29802f120ad31979d47be728b78b5f3f3d4d, review f2b70059, and result manifest f32df524d80c2aae46a0256ee762833fcbd32becaf59e00e901e17e4efcd4fe0. This includes all four rank journals, allocator/runtime receipts, eight original and sanitized planner reports, and synthetic adapter identity. Reports retain replay_complete=false: device token/layout inputs are absent, so native reproduction requires the pinned synthetic harness, adapter and runtime. All eight owned process identities were gone and all four GPUs were compute-empty at 8:38:02 AM Mountain. There was no automatic retry.

Additional September 24 diagnostic evidence from the Hayek/Schulman Codex investigation: matched cold execution OOMs; representative warmup leads to a planner refusal; separately forced warmed execution also OOMs. This resolves the earlier missing allocator-setting comparison for these new arms, without retroactively establishing the old run's allocator state.

All three arms used two H200s, CP2/EP2 and DP1/TP1/PP1/ETP1, BF16 Qwen3.6-35B-A3B revision 995ad96eacd98c81ed38be0c5b274b04031597b0, the same rebuilt runtime, and both allocator aliases set to expandable_segments:True with effective native settings verified. Same-rank comparisons passed for full input tensors, initial adapter/canonical state, absent optimizer state, and restored Python/NumPy/CPU/CUDA/Megatron RNG. The rebuilt runtime is not claimed binary-identical to the older four-GPU run.

The target preserves all 19 histories, 1,073,093 tokens and the original nested 10+9 callback. The first outer group has 779,067 tokens, two model slots and one unsplit subforward. Full lengths are:

94017, 93189, 89049, 84906, 80763, 76618, 72480, 68338, 64202,
55505, 51373, 47238, 43111, 38974, 32251, 28111, 23973, 16565, 12430

Oversized admission and the 10% planner-miss threshold were enabled. Representative warmup uses max(64, length // 4) for each of the same 19 histories: 268,265 tokens, two forward/backward waves and one optimizer update. Before the full target, adapter/optimizer/RNG state is restored while profiles and warmed runtime state remain.

Arm	Full-target admission	Result
Cold	Normal; 49,149,155,642 required versus 105,226,374,349 available bytes	Actual CUDA OOM during forward; no completed target wave/backward/update
Warmed	Refused; 195,975,011,970 required versus 89,634,049,229 available after cache recovery	No full-target execution or target peak measurement
Warmed, private diagnostic override	`admitted_oversized`, preserving the same plan and estimates	Actual CUDA OOM during forward; no completed target wave/backward/update

The ordinary warmed EP refusal was explicitly nonoverridable despite the global oversized flag. The third arm used a separately reviewed private helper that changed only that exact refusal's overridable flag; this was not a production admission change. Native cache recovery still ran, raising the world-reduced available budget from 49.144 GB to 89.634 GB. Its selected plan retained an earlier availability check because recovery replaces the best check only for a strictly lower requirement; that older field is not the post-recovery sample.

Cold rank1 failed at 11:58:10 AM Mountain in compiled expert-FC2 LoRA addition. Actual traceback locals and generated code establish a BF16 [2021662, 2048] output: 8,280,727,552 logical bytes. Its interrupted incremental allocated-peak lower bound was 102,710,431,744 bytes versus 44,681,050,584 predicted. This is not a completed peak.

Forced-warm rank0 failed at 12:19:24 PM Mountain on BF16 [1962245, 2048]: 8,037,355,520 logical bytes. The original CUDA exception reports a 7.49 GiB request, 6.60 GiB physically free, 121.37 GiB PyTorch allocated, and 1.30 GiB reserved but unused. Unused reservation does not establish a usable allocation; allocator-layout effects remain unresolved. Aggregate forward-entry hooks support failure in the second model slot's layer 22, but direct slot/layer identity was not captured in the exception frame. The sibling rank was interrupted, not independently classified as another OOM.

Reproduction assets and original generated failure kernels are retained in ~/.local/share/schulman/hayek-cold-warm-20260924/. Baseline source manifest: a34e5bcb79b09fe28661c8663f955df4298394dbadd5e4018e67594ca4489263; forced source manifest: 3fa112de04af696cf5202a4b1043fd00eae86c1d621daceb548ebe8d7b260dc1; forced execution: 15a259d1f334494c8b491344d89d72ef; forced OOM evidence: ed228eb22588c88b1b82a38379ed4ab15823b84dac30ef6ba0aea8ac861a74b3; final summary: ca0575383b9fa27403f305e71fd377669799cd794f4ec01e012c5a8aa653d9e1. These are synthetic reproduction assets, not production-token/checkpoint replay. All allocated processes and GPUs were released.

Warmup did not rescue this tested unsplit execution. These results do not establish a fundamental four-GPU minimum. A separate shortened per-history execution completed but failed its output, reduced-gradient and optimizer-update comparisons; its full-length successor was correctly withheld and is not an equivalent lower-memory replacement.

Apply trainer-rank memory floors per context-parallel rank (#951)

A related ordinary-admission failure is now independently verified in the retained September 25 full19 diagnostic. This is a modified, synthetic CP2 execution and a different failure site from the original issue; it is not a clean-current-main reproduction or proof of an identical root cause.

The first original group contains 10 requests / 779,067 packed tokens, two gradient-enabled slot groups (400,511 and 378,556). The selected outcome was admitted, required 49,149,155,642 bytes, world-reduced available 105,226,374,349 bytes, safety factor 1.1 and physical-free reserve 3%. Oversized admission was disabled. The original pre-execution checks are retained; these are not post-error estimates.

Forward completed for that first group. During layer-3 backward recomputation, rank 1 failed in HybridEP combine_with_unpermute at 3:12:40 AM Mountain with an 856 MiB request and 638.69 MiB reported free. The observer immediately before that call measured:

Contiguous FC1 output [1722026, 1024] BF16, 3,526,709,248 bytes, with its matched storage weak reference still alive.
Incoming combine tensor [1722026, 2048] BF16, 7,053,418,496 bytes.
Configured combine capacity P=218,752, H=2048: 896,008,192 bytes of output payload. Actual trimmed length is 218,751. Payload size and the allocator-rounded 856 MiB request remain distinct.
PyTorch allocated/reserved/physical-free immediately before the call: 137,467,786,752 / 139,359,944,704 / 667,615,232 bytes. This unsynchronized observation is distinct from the later OOM message.

Live storage is a concrete offload target, not proof of exclusive saved-tensor ownership, recoverable capacity, or full-step fit. No offload change was installed in this observation-only attempt. The second original group and optimizer update were not reached. Exact original OOM evidence precedes later owner cancellation of the unmatched-collective failure path; all registered/sampled owned identities and both GPUs were independently reconciled and released.

A source audit identifies an accounting domain gap: this full-recompute cold requirement is exactly int(1.1 * (779067 * 2048 * 2 * 14 + 2 * 779067 * 4)). The detailed routed-MoE floor returns zero outside TP1/CP1, including this CP2 case, while full recomputation uses the generic proxy. Consequently the cold formula has no explicit expanded expert M or HybridEP capacity term. The same floor/guard remains in inspected ART main 44bc5d4c; that source observation is not a current-main GPU reproduction. The first forward's retained incremental peaks already exceeded the cold estimate; backward did not finish and supplies no completed calibration. A bounded trainer-rank-only correction proposal and separately scoped offload tests are underway.

Durable evidence:

Source manifest: 19971194a9999acd931b6de90b91d872e1400aaafbcf0122d37c9b5864e2cafb.
Independent streamed audit: /home/brad/.local/share/schulman/successor-reviews-20260925/full19-instrument-attempt01-audit/REVIEW-RECEIPT.json, SHA256 de3680d477ec76bf4e75734579a3636e717efe6e595c176c9f60db72a317422a.
Original planner samples: adjacent PLANNER-RETAINED.json, SHA256 2e5ef15c23e3cfb31adb89a5cc4479765efe09b444aed96ae65344b19bc414da.
Source-term audit: /home/brad/.local/share/schulman/successor-reviews-20260925/full19-estimator-source-01/REVIEW-RECEIPT.json, SHA256 5297c8277aa2e43e16da2e3cf718eea97adc2ee1f9ac0116987182e79b81b915.
Root closure: 2dd64b87fa9aa2d8308b440aeeef5c2ff278d7f3b2864da0297ad016c60f6386.

Navigation Menu

Repository navigation

TrainerRank CP2/EP1 normal admission reaches CUDA OOM in compiled MoE LoRA forward #949

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TrainerRank CP2/EP1 normal admission reaches CUDA OOM in compiled MoE LoRA forward #949

Description

Activity

bradhilton commented on Sep 24, 2026

bradhilton commented on Sep 24, 2026

bradhilton commented on Sep 24, 2026

bradhilton commented on Sep 24, 2026

bradhilton commented on Sep 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions