Repository navigation
FP8 combine replies across NVLink domains (opt-in, stacked on #3) - #12
Merged
Merged
Conversation
AFDEP_HYBRID_FP8_COMBINE=1 makes FFN forwarders send each reduced reply row across NVLink domains as E4M3 with one UE8M0 power-of-two scale per 128 elements (H + align32(H/128) bytes instead of 2H; 4,128 instead of 8,256 at hidden 4096), and the Attention reduce epilogue dequantize each FFN domain's row and sum them in FP32 before one BF16 rounding. The kernels are copies generated by tests/role_kernels.py: the FFN hybrid combine and its sparse copy, combine_reduce with only its two shared-memory stores replaced by a quantizing store, and the reduce epilogue; afdep_fp8_combine.cuh holds the row format, the exact exponent rule and the dequantizing reduction. Shared kernels, scale-up partials, collects, signals, launch shapes and window sizing are unchanged; both FP8 regions keep their BF16 base addresses, so every FP8 byte lies inside the BF16 region it replaces. The native buffer latches the switch at construction (directional hybrid only) and reports it as a fifth element of the hybrid agreement tuple, which afdep/deepep.py checks on every rank and exposes as Buffer.fp8_combine. The GPU suite gains an FP8-aware bit-exact reference, a wide-range BF16 case and hidden 4096; docs/v1.md section 19 describes the draft, which has not run on GPU yet.
The forwarder now zeroes the bytes between the last scale byte and the 32-byte-aligned row end, so the reply row on the wire is deterministic and the epilogue's 4-byte scale loads never read stale shared memory. The top scale 2^120 cannot represent a group amax of 248 * 2^120 or more, which now decodes to infinity; the header, docs/v1.md section 19 and the exponent test state the limit.
AFDEP_HYBRID_FP8_COMBINE_MIN_BUCKET=b keeps BF16 replies for exchanges whose bucket is below b tokens. Below about 64 tokens the round trip is bound by latency, and the Attention dequantize epilogue adds about 4 us that the halved bytes do not pay back. Every rank passes the same bucket, so both roles pick the same row format. The value is latched with the switch, joins the hybrid agreement tuple, and is reported as Buffer.fp8_combine_min_bucket. The GPU suite's reference follows each exchange's bucket, so its bucket cycle covers format changes between back-to-back exchanges.
The combine layout assert again checks the latched switch, so a flipped AFDEP_UNION_HYBRID or a weighted combine fails at every bucket rather than only at FP8 ones. docs/v1.md section 19 and docs/api.md now give the minimum's default and clamping, the six-element agreement tuple, and why a format change between back-to-back exchanges keeps the protocol intact; the build test pins the latch, the clamp and the assert's position.
docs/validation-v1-fp8-combine.json records the GPU suite on 024e98b0 and on the gated 2cee2f51, deferred-put fault injection with negative controls, paired transport screens on 4-GPU and 8-GPU NVLink domains, the kernel trace behind the small-bucket cost, and a 16A:8F serving A/B, all on kermit H200 over InfiniBand. docs/v1.md section 19, docs/api.md and the README now state the measured effect instead of the draft estimate; the accuracy check is still open.
Brings in the design doc and record renames and the cross-node records' new names. Only LOCAL_CHANGES.json conflicted, on the hashes of buffer.hpp and afdep_suites.hpp, which both sides edited; the scope keeps main's reworded entries and the FP8 entry. The FP8 additions' references to docs/v1.md in the README, the FP8 record, the comm benchmark and two test comments now name docs/design.md; the FP8 record's own name and wording change in the next commit. The merged edits to buffer.hpp and afdep_suites.hpp change only comments; LOCAL_CHANGES.json carries the refreshed hashes and the native source digest is now b5078d62.
docs/validation-v1-fp8-combine.json is now docs/validation-fp8-combine.json, and the README's FP8 paragraph and section 19 of the design doc link to it. In its screen tables the arms without the sparse switches are default and default_fp8 (with default_fp8_paired_pct) instead of v1 and v1_fp8; numbers and digests are unchanged. The native tree is unchanged (b5078d62), and the figures regenerate identically.
GSM8K test (1,319 problems, zero-shot, greedy, thinking off) on Qwen3-235B served by vLLM with AFD 16A:8F and the sparse switches, on tree b5078d62. FP8 replies at every bucket score 95.07% against 95.30% and 95.07% for two BF16 runs (McNemar p = 0.75 and 1.0). Per-layer MoE error and next-token KL stay open.
Etelis
marked this pull request as ready for review
October 24, 2026 12:26
3 of 6 tasks
Etelis
added a commit
that referenced
this pull request
Oct 25, 2026
## What and why Sections 17 to 19 (sparse and mixed-role directional hybrids, FP8 combine replies) were merged as opt-in additions in #3 and #12, but the README and `docs/design.md` still described them as a proposal under review on the cross-node branch. - README: the section "Cross-node proposal (opt-in, under review)" becomes "Cross-node additions (opt-in)". The two links to its anchor and the numbers summary are updated to match. - `docs/design.md`: the heading "Proposed (opt-in, under review)" becomes "Opt-in additions after 1.0.0". It now says the sections were merged after the 1.0.0 release (#3 and #12) and that 1.0.0 does not ship them. The decisions in sections 17 and 19 now say "merged as opt-in, off by default" instead of "opt-in until reviewed", "needs a maintainer decision" and "flagged for review". Docs only. Defaults, code and the dated validation records are unchanged. ## Checks - [x] `python -m pytest tests/unit`, `ruff check .` and `ruff format --check .` pass - [ ] Native changes: none - [ ] Performance claims: none - [x] Records and docs/design.md name the exact native source digest measured (unchanged) - [ ] Optimizations are AFD-specific and keep a control switch: not applicable - [x] Communication-only results are labeled as such; no serving claims without serving evidence ## Evidence The link check covers relative Markdown links, `src`/`srcset`, heading anchors, and every `docs/...` path in tracked text files: 238 references, none broken.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Opt-in FP8 combine replies across NVLink domains on the directional hybrid (
AFDEP_HYBRID_FP8_COMBINE=1, docs/design.md §19), built on the cross-node proposal (#3).In the directional hybrid every reply byte of S Attention domains leaves through the few NICs of the FFN domain, and BF16 replies are two thirds of a token's round-trip bytes. With the switch, FFN forwarders quantize each reduced reply row to E4M3 with one power-of-two scale per 128 elements (4,128 instead of 8,256 bytes at hidden 4096), and the Attention reduce epilogue dequantizes and sums the FFN domains in FP32.
AFDEP_HYBRID_FP8_COMBINE_MIN_BUCKET=bkeeps BF16 replies for buckets below b tokens, where the round trip is latency-bound and the dequantize epilogue's fixed ~4 µs outweighs the bytes saved; every rank passes the same bucket, so both roles agree on the format. The switch and the minimum are latched at construction and checked on every rank. Scale-up partials, collects, signals and protocol words are unchanged, and both FP8 regions keep their BF16 base addresses. The kernels are generated copies (tests/role_kernels.py); the shared kernels stay byte-identical.Commits:
then a merge of the cross-node branch, the record renames, and:
Checks
python -m pytest tests/unit,ruff check .andruff format --check .pass (47 tests)fp8_range), top-k 2, every sparse switch combination, two FFN domains, one Attention domain, the shared collect, and minimum buckets inside the capacity so the bucket cycle switches format both ways in one Buffer and one graph; off topologies report the switch off; a role-local mismatch is rejected on every rank; deferred-put fault injection passes with negative controls that fail024e98b0ungated,2cee2f51gated,b5078d62for the accuracy check)Evidence
Record:
docs/validation-fp8-combine.json. Round trip against the sparse switches, hidden 4096, FP8 dispatch, with two or more Attention domains: −3% to −5% at 64 tokens per Attention GPU, −13% to −21% at 512; with one Attention domain −10% to −11% at 512. With the minimum bucket at 64, buckets below it stay within 0.6% (4F:12A) and 1.8% (8-GPU domains) of BF16. Serving (vLLM, Qwen3-235B, 4,096 in / 2,048 out, 16 8000 A:8F): TPOT −1.4% to +0.9%; at 131K context with 52A:8F, decode is unchanged (its buckets stay below 64) and median time to first token drops 7% (217 → 202 s). Accuracy (treeb5078d62, comments-only native changes since2cee2f51): GSM8K test, 1,319 problems, zero-shot, greedy, on Qwen3-235B served by vLLM with 16A:8F and the sparse switches. FP8 replies at every bucket score 95.07% against 95.30% and 95.07% for two BF16 runs (McNemar p = 0.75 and 1.0; 95% interval for the loss about −0.7 to +1.2 points). Per-layer MoE error and next-token KL are not measured yet, so it stays opt-in and off by default.