Repository navigation
Cross-node AFD: sparse and mixed-role directional hybrids (opt-in) - #3
Merged
Merged
Conversation
With one FFN NVLink domain and S Attention domains, the directional hybrid's per-channel signals scale with the channel count C, not the tokens: every source sends C finish tails into the FFN NIC, the FFN signals every source on every channel in combine, and every source's token 0 rides channel 0. Three opt-in switches, through kernels generated by tests/role_kernels.py (shared kernels and defaults unchanged): - AFDEP_HYBRID_STAGGER=1: source s puts token i on channel (i + s*C/S) mod C (AFDEP_HYBRID_STAGGER_MAX_TOKENS bounds it; default any). - AFDEP_HYBRID_SPARSE_DISPATCH=1: sources publish their token count and send finish tails only on used channels; FFN forwarders retire the rest. - AFDEP_HYBRID_SPARSE_COMBINE=1: the FFN signals a source only on channels that replied to it, plus channel 0 as a done witness; the collect waits for exactly those, from a mask its dispatch recorded. afdep/deepep.py rejects ranks that resolve the switches differently. 12 pods x 2 H200 (11:1, FP8, hidden 2048, top-8 of 128, C = 192), paired against v1 (958652a3), final tree 200944ce with all three: -68.1% at 1 token, -66.8% at 8, -62.3% at 16, -50.7% at 32, -26.7% at 64, -7.1% at 128, -3.0% at 256, +0.1% at 512 (Masked, earlier unpaired run: -78% at 1 token). Sparse combine alone gains 2.4%; the switches are not additive (section 17). GPU suite on 24 ranks passes for every configuration tried; a collect that waits only for channel 0 fails it; a mismatched FFN-only shape control is rejected on every rank. Unit tests 41/41, Ruff clean. Record: docs/validation-v1-sparse-hybrid.json.
…mains On 200944ce, all three switches, paired against v1 (2 repeats, every repeat faster): two FFN domains (4 FFN + 20 Attention ranks) -67.1% at 1 token to -8.6% at 128 and -1.7% at 256; three Attention domains (2 FFN + 6 Attention ranks, 3:1) -33.9% at 1 token, -35.7% at 8, -6.0% at 128. The gain scales with the number of Attention domains. The GPU suite passes with two FFN domains at capacities 8 and 64 on 24 ranks.
The construction-time agreement check also ran in one NVLink domain and in mixed-role layouts, where the switches do nothing, so ranks that set them, or the role-local AFDEP_SHARED_* controls they depend on, differently were rejected there. get_afd_sparse_hybrid_config now reports the switches off outside the directional hybrid (found by an independent review of in-node invariance). On 2d22e5e9: unit tests 41/41; GPU suite on kermit, 6 pods x 4 H200 (4 FFN + 20 Attention), all three switches, 24 ranks; the FFN-only AFDEP_SHARED_HYBRID_FORWARD=1 still rejected there on every pod; and in one NVLink domain (8 H200) the same control with every switch set now runs the suite normally. Also recorded: in-node screens (6A:2F, 7A:1F, 4A:4F, 1 to 2,048 tokens, 200944ce against 958652a3), switches off -0.8..+0.8% and on -1.2..+1.0%; kermit single-role 5:1, all three switches -48.8% at 1 token, -50.5% at 8, -40.8% at 32; kermit mixed-role 11:1, where the role path runs the union hybrid and matches Masked, with 8 SMs -17% at 1 token but +56% at 512.
The channel-0 witness of the sparse combine (section 17) followed channel 0's own forward warp only. A source waits for its own channels and channel 0, so it could start its next dispatch, whose payload lands in the rail forwarder's dispatch scale-out inbox, while other forward warps still read the forwarder's combine scale-up inbox, which can overlap it at FP8 (found by an independent design review). Now every forward warp releases a rank-local DONE count after its last read, and channel 0 waits for all C before the witness. The GPU suite gains a one-small-source case with spread routing (a token's experts on every FFN rank). On kermit 4F:20A (FP8, hidden 2048, capacity 2048, 24 SMs, all three switches) the fixed tree passes with and without forward warps delayed ~200 us. Three fault runs did not make the unfixed tree fail, so the gap is argued from the layout and the code, not reproduced. Cost, paired (2 repeats): +6.5% (5 us) at 8 tokens, within 1.2% at 1, 32, 128 and 512; still -47% against v1 at 8 tokens.
At high Attention:FFN ratios the FFN group is smaller than a node, so FFN ranks share their NVLink domain with Attention ranks and the role path falls back to the union hybrid, which matches Masked. With AFDEP_MIXED_HYBRID=1 a layout with FFN ranks [0, F), 0 < F < U, runs the directional protocol with one FFN-holding domain and roles per rank, through generated copies of hybrid_dispatch.cuh and hybrid_combine.cuh: every domain-0 GPU stays its rail's forwarder (node-wide ingress), only ranks below F hold experts, domain-0 Attention GPUs also send (local bypass, self counts) and reply to themselves locally, and pure-Attention domains keep the send block and the collect. Every rank's reply inbox moves into a slice reserved before the combine staging slice, and dispatch and combine assert both layouts end below the slices. The agreement tuple carries F. Shared kernels and defaults are unchanged. kermit 6 x 4 H200, FP8, hidden 2048, top-8, capacity 2048, paired against the v1 union role path (2 repeats, every repeat faster), on 81b71eb0: 11:1 (F=2, 128 experts) -23.2% at 1 token, -19.7% at 8, -21.7% at 32, -14.8% at 128, -8.1% at 512; 23:1 (F=1, 64 experts) -20.7% to -6.2%. Masked matches union. GPU suite on 24 ranks passes at F=2 (FP8 and BF16, capacities 8 and 64), F=1 and F=3. Unit tests 41/41, Ruff clean. Not yet: the sparse switches on this path, fault injection for the reply-slice and staging orderings, a two-buffer interleave.
Sections 17 and 18 move under a Proposed heading. Every number names the tree it was measured on: the 12-domain and 5:1 screens predate the done witness, which was screened and suite-run only on kermit 4F:20A. The records map each measured tree to its published digest, rename the control arm to v1, mark the channel-0-only witness as the unsafe earlier tree, and drop a harness-only variable. The docs add that the switches are read at every launch but checked once at construction, that the sparse path was run with FP8 only, and that the mixed path was validated on 4-GPU NVLink domains only.
A section for the two opt-in additions, with three figures: the sparse switches at 11:1 over 12 domains, the sparse hybrid at other layouts, and the mixed-role path. Every number names the tree it was measured on; the 12-domain screens predate the done witness. The support section states the narrower validation of the proposal.
4 of 6 tasks
docs/v1.md is now docs/design.md, and each docs/validation-v1-<name>.json record is docs/validation-<name>.json. Links in the README, docs and templates, cross-references inside the records, the figure data's sources and two comments in afdep_suites.hpp follow. Section headings are unchanged, so every anchor still resolves. The comment edits change only comments; LOCAL_CHANGES.json carries the refreshed file hash.
The README, design doc, notices, provenance text and records call the role path afdEP, the role path or the release instead of v1. The design doc's naming note becomes a digest note: 958652a3 is the tree of commit 32c871d, and the commits after it changed only comments, notices and other text in the native tree (the 1.0.0 release tree is 131efc37). The 8 x H200 and cross-node record names its role arm afd, as the other records do, and the role-suites record names the tree before the suites pre_suites. Tests call the pinned shared-kernel hashes SHARED_KERNEL_SHA256 and drop v1 from a test name. The comment edit in buffer.hpp and the provenance text change only comments and text; the native source digest is now db115728.
benchmarks.json names its role series afdEP, and plot_benchmarks.py uses that label in its legends and subtitles. The regenerated result figures differ from the previous ones only in that text and in the legend's second entry, which moves left by the width of the dropped " v1". The two hand-drawn diagrams change only their title, description and heading text.
docs/v1.md becomes docs/design.md with the cross-node sections 17 and 18, and main's records take their new names. The conflicts keep the cross-node text: the README's links to sections 17 and 18 point at docs/design.md, docs/api.md keeps the opt-in protocols, the design doc's digest note keeps the published digests of the cross-node trees, and LOCAL_CHANGES.json keeps the sparse and mixed-role scope entries after main's reworded ones. The figure script names its role series afdEP, so the cross-node proposal data in benchmarks.json does too, and the regenerated results-xn figures differ only in that label, the subtitles and the legend entries after it, which move left by the width of the dropped " v1". The two cross-node records cite docs/design.md; their own names and wording change in the next commit. The merged comment edits in buffer.hpp and afdep_suites.hpp change only comments; LOCAL_CHANGES.json carries the refreshed hashes and the native source digest is now 233548d8.
The sparse and mixed-role records are now docs/validation-sparse-hybrid.json and docs/validation-mixed-hybrid.json; the README, the design doc and the mixed-role record's cross-reference follow. The arm the proposal is paired against, the role path on 958652a3 with no sparse switches, is the default in prose and tables: the records' v1 arms and order entries are default, control_v1 is control_default, and the README rows say afdEP as the figures do. The records' notes on published trees no longer name the generation. Numbers and digests are unchanged, and the figures regenerate identically. The native tree is unchanged (233548d8).
3 of 6 tasks
Etelis
added a commit
that referenced
this pull request
Oct 25, 2026
## What and why Sections 17 to 19 (sparse and mixed-role directional hybrids, FP8 combine replies) were merged as opt-in additions in #3 and #12, but the README and `docs/design.md` still described them as a proposal under review on the cross-node branch. - README: the section "Cross-node proposal (opt-in, under review)" becomes "Cross-node additions (opt-in)". The two links to its anchor and the numbers summary are updated to match. - `docs/design.md`: the heading "Proposed (opt-in, under review)" becomes "Opt-in additions after 1.0.0". It now says the sections were merged after the 1.0.0 release (#3 and #12) and that 1.0.0 does not ship them. The decisions in sections 17 and 19 now say "merged as opt-in, off by default" instead of "opt-in until reviewed", "needs a maintainer decision" and "flagged for review". Docs only. Defaults, code and the dated validation records are unchanged. ## Checks - [x] `python -m pytest tests/unit`, `ruff check .` and `ruff format --check .` pass - [ ] Native changes: none - [ ] Performance claims: none - [x] Records and docs/design.md name the exact native source digest measured (unchanged) - [ ] Optimizations are AFD-specific and keep a control switch: not applicable - [x] Communication-only results are labeled as such; no serving claims without serving evidence ## Evidence The link check covers relative Markdown links, `src`/`srcset`, heading anchors, and every `docs/...` path in tracked text files: 238 references, none broken.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two opt-in additions for afdEP across NVLink domains (InfiniBand, NCCL GIN). With every switch off, the default role path's kernels and launches are unchanged, and in one NVLink domain nothing changes.
AFDEP_HYBRID_STAGGER=1,AFDEP_HYBRID_SPARSE_DISPATCH=1,AFDEP_HYBRID_SPARSE_COMBINE=1). The default directional protocol signals all C channels (192 at 24 SMs) every exchange, whether they carry tokens or not; at 11:1 an FFN GPU's NIC takes 2,112 finish atomics in dispatch and sends as many combine signals. The switches stagger each source's first channel, publish token counts so FFN forwarders retire unused channels, and signal only the channels that replied, plus a done witness sent once every forward warp of the forwarder has finished.AFDEP_MIXED_HYBRID=1, prototype). When the FFN GPUs share a node with Attention GPUs, afdEP falls back to the union hybrid and matches Masked DeepEP within 1.4%. This gives every GPU its role individually: each GPU on the FFN node still receives its rail's tokens (node-wide ingress), only FFN GPUs hold experts, the node's Attention GPUs also send locally, and Attention-only nodes just send and collect.New behaviour runs through kernels generated by
tests/role_kernels.pyfrom the shared DeepEP V2 kernels, which stay byte-identical (pinned by a unit test). Every switch must be set alike on all ranks before theBufferis constructed; construction rejects a mismatch.The commits are the five original development commits in order, then a review-fix commit and the README, then merges of main and the record renames.
Results
Communication only: round trip under CUDA graphs, H200, max over Attention ranks; paired geometric-mean change against the default role path (
958652a3) over fresh-process repeats.200944ce200944ce200944ce2d22e5e981b71eb081b71eb0The sparse rows predate the done witness. The tree with it (
7b8ed8b5) was screened at 5:1 only: +6.6% at 8 tokens and within ±1.2% elsewhere against2d22e5e9. The 12-domain layouts have not been re-measured with it. The measured trees differ from the published ones only in comments and text;docs/design.mdand the records map each to its published digest (branch head233548d8).Figures, full tables and the in-node invariance screens are in the README. Design, correctness arguments and records are in
docs/design.mdsections 17 and 18 and indocs/validation-sparse-hybrid.json/docs/validation-mixed-hybrid.json.Validation
7b8ed8b5), only on 6 × 4 H200 at capacity 2,048, with and without a fault-injection delay. A collect that waits only for channel 0 fails the suite (negative control).200944cematches the default role path within ±1.2% with the switches off and on. The done witness and the mixed-role path run only with two or more NVLink domains.Known limits
Bufferexists bypasses the check.Follow-ups
Open evidence and follow-ups are tracked in #10. Latching the switches at construction is #7, and the two-buffer interleave of the mixed-role path depends on #5.