8000 FP8 combine replies across NVLink domains (opt-in, stacked on #3) by Etelis · Pull Request #12 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content

FP8 combine replies across NVLink domains (opt-in, stacked on #3) - #12

Merged
Etelis merged 8 commits into
mainfrom
fp8-combine
Oct 24, 2026
Merged

Etelis merged 8 commits into
mainfrom
fp8-combine

Conversation

@Etelis
@Etelis Etelis commented Oct 24, 2026 •
Copy link
Copy Markdown
Member

What and why

Opt-in FP8 combine replies across NVLink domains on the directional hybrid (AFDEP_HYBRID_FP8_COMBINE=1, docs/design.md §19), built on the cross-node proposal (#3).

In the directional hybrid every reply byte of S Attention domains leaves through the few NICs of the FFN domain, and BF16 replies are two thirds of a token's round-trip bytes. With the switch, FFN forwarders quantize each reduced reply row to E4M3 with one power-of-two scale per 128 elements (4,128 instead of 8,256 bytes at hidden 4096), and the Attention reduce epilogue dequantizes and sums the FFN domains in FP32. AFDEP_HYBRID_FP8_COMBINE_MIN_BUCKET=b keeps BF16 replies for buckets below b tokens, where the round trip is latency-bound and the dequantize epilogue's fixed ~4 µs outweighs the bytes saved; every rank passes the same bucket, so both roles agree on the format. The switch and the minimum are latched at construction and checked on every rank. Scale-up partials, collects, signals and protocol words are unchanged, and both FP8 regions keep their BF16 base addresses. The kernels are generated copies (tests/role_kernels.py); the shared kernels stay byte-identical.

Commits:

  • Add opt-in FP8 combine replies for the cross-node directional hybrid
  • Zero the FP8 reply row's scale padding and document the top scale
  • Gate FP8 combine replies on a minimum bucket
  • Keep the FP8 layout assert at every bucket and document the minimum
  • Record the FP8 combine replies on H200

then a merge of the cross-node branch, the record renames, and:

  • Record a GSM8K accuracy check of the FP8 combine replies

Checks

  • python -m pytest tests/unit, ruff check . and ruff format --check . pass (47 tests)
  • Native changes: the GPU suite on H200 over InfiniBand (6 × 4 and 4 × 4 GPUs): hidden 2048/4096/7168/8192, FP8 and BF16 dispatch (with fp8_range), top-k 2, every sparse switch combination, two FFN domains, one Attention domain, the shared collect, and minimum buckets inside the capacity so the bucket cycle switches format both ways in one Buffer and one graph; off topologies report the switch off; a role-local mismatch is rejected on every rank; deferred-put fault injection passes with negative controls that fail
  • Performance claims: paired fresh-process screens against the change's own control switch on the same tree (sparse vs sparse + FP8, default vs default + FP8) on 4-GPU and 8-GPU NVLink domains; no Masked DeepEP arm in these screens (Masked was screened on Cross-node AFD: sparse and mixed-role directional hybrids (opt-in) #3)
  • Records and docs/design.md name the exact native source digests (024e98b0 ungated, 2cee2f51 gated, b5078d62 for the accuracy check)
  • Optimizations are AFD-specific and keep a control switch: it keeps a control switch, but §19 notes that the byte cut would also shrink Masked's combine and that this is the library's first quantization step, so it needs a maintainer decision
  • Communication-only results are labeled as such; the serving numbers come from vLLM serving runs

Evidence

Record: docs/validation-fp8-combine.json. Round trip against the sparse switches, hidden 4096, FP8 dispatch, with two or more Attention domains: −3% to −5% at 64 tokens per Attention GPU, −13% to −21% at 512; with one Attention domain −10% to −11% at 512. With the minimum bucket at 64, buckets below it stay within 0.6% (4F:12A) and 1.8% (8-GPU domains) of BF16. Serving (vLLM, Qwen3-235B, 4,096 in / 2,048 out, 16 8000 A:8F): TPOT −1.4% to +0.9%; at 131K context with 52A:8F, decode is unchanged (its buckets stay below 64) and median time to first token drops 7% (217 → 202 s). Accuracy (tree b5078d62, comments-only native changes since 2cee2f51): GSM8K test, 1,319 problems, zero-shot, greedy, on Qwen3-235B served by vLLM with 16A:8F and the sparse switches. FP8 replies at every bucket score 95.07% against 95.30% and 95.07% for two BF16 runs (McNemar p = 0.75 and 1.0; 95% interval for the loss about −0.7 to +1.2 points). Per-layer MoE error and next-token KL are not measured yet, so it stays opt-in and off by default.

AFDEP_HYBRID_FP8_COMBINE=1 makes FFN forwarders send each reduced reply
row across NVLink domains as E4M3 with one UE8M0 power-of-two scale per
128 elements (H + align32(H/128) bytes instead of 2H; 4,128 instead of
8,256 at hidden 4096), and the Attention reduce epilogue dequantize each
FFN domain's row and sum them in FP32 before one BF16 rounding.

The kernels are copies generated by tests/role_kernels.py: the FFN hybrid
combine and its sparse copy, combine_reduce with only its two
shared-memory stores replaced by a quantizing store, and the reduce
epilogue; afdep_fp8_combine.cuh holds the row format, the exact exponent
rule and the dequantizing reduction. Shared kernels, scale-up partials,
collects, signals, launch shapes and window sizing are unchanged; both
FP8 regions keep their BF16 base addresses, so every FP8 byte lies
inside the BF16 region it replaces.

The native buffer latches the switch at construction (directional
hybrid only) and reports it as a fifth element of the hybrid agreement
tuple, which afdep/deepep.py checks on every rank and exposes as
Buffer.fp8_combine. The GPU suite gains an FP8-aware bit-exact reference,
a wide-range BF16 case and hidden 4096; docs/v1.md section 19 describes
the draft, which has not run on GPU yet.
The forwarder now zeroes the bytes between the last scale byte and the
32-byte-aligned row end, so the reply row on the wire is deterministic and
the epilogue's 4-byte scale loads never read stale shared memory. The top
scale 2^120 cannot represent a group amax of 248 * 2^120 or more, which now
decodes to infinity; the header, docs/v1.md section 19 and the exponent test
state the limit.
AFDEP_HYBRID_FP8_COMBINE_MIN_BUCKET=b keeps BF16 replies for exchanges whose
bucket is below b tokens. Below about 64 tokens the round trip is bound by
latency, and the Attention dequantize epilogue adds about 4 us that the
halved bytes do not pay back. Every rank passes the same bucket, so both
roles pick the same row format. The value is latched with the switch,
joins the hybrid agreement tuple, and is reported as
Buffer.fp8_combine_min_bucket. The GPU suite's reference follows each
exchange's bucket, so its bucket cycle covers format changes between
back-to-back exchanges.
The combine layout assert again checks the latched switch, so a flipped
AFDEP_UNION_HYBRID or a weighted combine fails at every bucket rather than
only at FP8 ones. docs/v1.md section 19 and docs/api.md now give the
minimum's default and clamping, the six-element agreement tuple, and why a
format change between back-to-back exchanges keeps the protocol intact; the
build test pins the latch, the clamp and the assert's position.
docs/validation-v1-fp8-combine.json records the GPU suite on 024e98b0 and
on the gated 2cee2f51, deferred-put fault injection with negative controls,
paired transport screens on 4-GPU and 8-GPU NVLink domains, the kernel
trace behind the small-bucket cost, and a 16A:8F serving A/B, all on kermit
H200 over InfiniBand. docs/v1.md section 19, docs/api.md and the README now
state the measured effect instead of the draft estimate; the accuracy check
is still open.
Brings in the design doc and record renames and the cross-node
records' new names. Only LOCAL_CHANGES.json conflicted, on the hashes
of buffer.hpp and afdep_suites.hpp, which both sides edited; the scope
keeps main's reworded entries and the FP8 entry. The FP8 additions'
references to docs/v1.md in the README, the FP8 record, the comm
benchmark and two test comments now name docs/design.md; the FP8
record's own name and wording change in the next commit.

The merged edits to buffer.hpp and afdep_suites.hpp change only
comments; LOCAL_CHANGES.json carries the refreshed hashes and the
native source digest is now b5078d62.
docs/validation-v1-fp8-combine.json is now docs/validation-fp8-combine.json,
and the README's FP8 paragraph and section 19 of the design doc link to
it. In its screen tables the arms without the sparse switches are
default and default_fp8 (with default_fp8_paired_pct) instead of v1 and
v1_fp8; numbers and digests are unchanged. The native tree is unchanged
(b5078d62), and the figures regenerate identically.
GSM8K test (1,319 problems, zero-shot, greedy, thinking off) on
Qwen3-235B served by vLLM with AFD 16A:8F and the sparse switches, on
tree b5078d62. FP8 replies at every bucket score 95.07% against 95.30%
and 95.07% for two BF16 runs (McNemar p = 0.75 and 1.0). Per-layer
MoE error and next-token KL stay open.
Base automatically changed from cross-node to main October 24, 2026 12:26
@Etelis
Etelis marked this pull request as ready for review October 24, 2026 12:26
@Etelis
Etelis requested a review from almogtavor as a code owner October 24, 2026 12:26
@Etelis
Etelis merged commit cfe8294 into main Oct 24, 2026
4 checks passed
@Etelis
Etelis deleted the fp8-combine branch October 24, 2026 12:26
@Etelis Etelis mentioned this pull request Oct 24, 2026
3 of 6 tasks
Etelis added a commit that referenced this pull request Oct 25, 2026
## What and why

Sections 17 to 19 (sparse and mixed-role directional hybrids, FP8
combine replies) were merged as opt-in additions in #3 and #12, but the
README and `docs/design.md` still described them as a proposal under
review on the cross-node branch.

- README: the section "Cross-node proposal (opt-in, under review)"
becomes "Cross-node additions (opt-in)". The two links to its anchor and
the numbers summary are updated to match.
- `docs/design.md`: the heading "Proposed (opt-in, under review)"
becomes "Opt-in additions after 1.0.0". It now says the sections were
merged after the 1.0.0 release (#3 and #12) and that 1.0.0 does not ship
them. The decisions in sections 17 and 19 now say "merged as opt-in, off
by default" instead of "opt-in until reviewed", "needs a maintainer
decision" and "flagged for review".

Docs only. Defaults, code and the dated validation records are
unchanged.

## Checks

- [x] `python -m pytest tests/unit`, `ruff check .` and `ruff format
--check .` pass
- [ ] Native changes: none
- [ ] Performance claims: none
- [x] Records and docs/design.md name the exact native source digest
measured (unchanged)
- [ ] Optimizations are AFD-specific and keep a control switch: not
applicable
- [x] Communication-only results are labeled as such; no serving claims
without serving evidence

## Evidence

The link check covers relative Markdown links, `src`/`srcset`, heading
anchors, and every `docs/...` path in tracked text files: 238
references, none broken.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0