8000 Cross-node AFD: sparse and mixed-role directional hybrids (opt-in) by Etelis · Pull Request #3 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content

Cross-node AFD: sparse and mixed-role directional hybrids (opt-in) - #3

Merged
Etelis merged 13 commits into
mainfrom
cross-node
Oct 24, 2026
Merged

Etelis merged 13 commits into
mainfrom
cross-node

Conversation

@Etelis
@Etelis Etelis commented Oct 23, 2026 •
Copy link
Copy Markdown
Member

Summary

Two opt-in additions for afdEP across NVLink domains (InfiniBand, NCCL GIN). With every switch off, the default role path's kernels and launches are unchanged, and in one NVLink domain nothing changes.

  1. Sparse directional hybrid (AFDEP_HYBRID_STAGGER=1, AFDEP_HYBRID_SPARSE_DISPATCH=1, AFDEP_HYBRID_SPARSE_COMBINE=1). The default directional protocol signals all C channels (192 at 24 SMs) every exchange, whether they carry tokens or not; at 11:1 an FFN GPU's NIC takes 2,112 finish atomics in dispatch and sends as many combine signals. The switches stagger each source's first channel, publish token counts so FFN forwarders retire unused channels, and signal only the channels that replied, plus a done witness sent once every forward warp of the forwarder has finished.
  2. Mixed-role directional hybrid (AFDEP_MIXED_HYBRID=1, prototype). When the FFN GPUs share a node with Attention GPUs, afdEP falls back to the union hybrid and matches Masked DeepEP within 1.4%. This gives every GPU its role individually: each GPU on the FFN node still receives its rail's tokens (node-wide ingress), only FFN GPUs hold experts, the node's Attention GPUs also send locally, and Attention-only nodes just send and collect.

New behaviour runs through kernels generated by tests/role_kernels.py from the shared DeepEP V2 kernels, which stay byte-identical (pinned by a unit test). Every switch must be set alike on all ranks before the Buffer is constructed; construction rejects a mismatch.

The commits are the five original development commits in order, then a review-fix commit and the README, then merges of main and the record renames.

Results

Communication only: round trip under CUDA graphs, H200, max over Attention ranks; paired geometric-mean change against the default role path (958652a3) over fresh-process repeats.

Layout Tree 1 token 32 128 512
Sparse, 11:1 over 12 single-role domains (12 × 2 GPUs) 200944ce −68% −51% −7% 0%
Sparse, two FFN domains (4F:20A, 12 × 2) 200944ce −67% −47% −9%
Sparse, three Attention domains (2F:6A, 4 × 2) 200944ce −34% −29% −6%
Sparse, 5:1 over six 4-GPU domains 2d22e5e9 −49% −41% −2% 0%
Mixed-role, 11:1 (FFN domain = 2 FFN + 2 Attention GPUs) 81b71eb0 −23% −22% −15% −8%
Mixed-role, 23:1 (64 experts) 81b71eb0 −21% −17% −12% −6%

The sparse rows predate the done witness. The tree with it (7b8ed8b5) was screened at 5:1 only: +6.6% at 8 tokens and within ±1.2% elsewhere against 2d22e5e9. The 12-domain layouts have not been re-measured with it. The measured trees differ from the published ones only in comments and text; docs/design.md and the records map each to its published digest (branch head 233548d8).

Figures, full tables and the in-node invariance screens are in the README. Design, correctness arguments and records are in docs/design.md sections 17 and 18 and in docs/validation-sparse-hybrid.json / docs/validation-mixed-hybrid.json.

Validation

  • Sparse switches, GPU suite (tests.gpu.deepep) on 24 ranks, FP8 only: 12 × 2 H200 with one and two FFN domains at capacities 8 and 64, and 6 × 4 H200 at capacity 8. Cases include empty, inactive, skewed, all-skipped and one-small sources, delayed peers and graph replay. With the done witness (7b8ed8b5), only on 6 × 4 H200 at capacity 2,048, with and without a fault-injection delay. A collect that waits only for channel 0 fails the suite (negative control).
  • Mixed-role, GPU suite on 6 × 4 H200 with F = 1, 2 and 3; FP8 at capacity 8 and BF16 at capacities 8 and 64.
  • In one NVLink domain (8 × H200, three layouts, 1 to 2,048 tokens), tree 200944ce matches the default role path within ±1.2% with the switches off and on. The done witness and the mixed-role path run only with two or more NVLink domains.
  • Unit tests 41/41, Ruff clean, CPU checks and distribution check pass.

Known limits

  • The done witness costs +6.6% at 8 tokens at 5:1. A zero-cost alternative, moving the dispatch scale-out inbox past the combine scale-up inbox, is not done yet. The race it closes is argued from the layout and the code; three fault-injection runs did not reproduce it.
  • The switches are checked once at construction but read at every launch: changing them after the Buffer exists bypasses the check.
  • The sparse switches were run with FP8 only and do not apply to the mixed-role path yet.
  • The mixed-role path is validated with one FFN-holding domain on 4-GPU domains. 8-GPU nodes, fault injection for its reply and staging slices, and a two-buffer interleave are not yet run. Its FFN-node Attention GPUs join that node's scale-up barriers, so with two buffers every GPU there must enqueue both buffers' exchanges in one order, as the union hybrid already requires.
  • Whether the sparse idea would also speed up Masked DeepEP's union hybrid is not measured (see the note in section 17).
  • No serving claims: these are communication-only measurements.

Follow-ups

Open evidence and follow-ups are tracked in #10. Latching the switches at construction is #7, and the two-buffer interleave of the mixed-role path depends on #5.

With one FFN NVLink domain and S Attention domains, the directional hybrid's
per-channel signals scale with the channel count C, not the tokens: every
source sends C finish tails into the FFN NIC, the FFN signals every source on
every channel in combine, and every source's token 0 rides channel 0. Three
opt-in switches, through kernels generated by tests/role_kernels.py (shared
kernels and defaults unchanged):

- AFDEP_HYBRID_STAGGER=1: source s puts token i on channel (i + s*C/S) mod C
  (AFDEP_HYBRID_STAGGER_MAX_TOKENS bounds it; default any).
- AFDEP_HYBRID_SPARSE_DISPATCH=1: sources publish their token count and send
  finish tails only on used channels; FFN forwarders retire the rest.
- AFDEP_HYBRID_SPARSE_COMBINE=1: the FFN signals a source only on channels
  that replied to it, plus channel 0 as a done witness; the collect waits for
  exactly those, from a mask its dispatch recorded.

afdep/deepep.py rejects ranks that resolve the switches differently.

12 pods x 2 H200 (11:1, FP8, hidden 2048, top-8 of 128, C = 192), paired
against v1 (958652a3), final tree 200944ce with all three: -68.1% at 1
token, -66.8% at 8, -62.3% at 16, -50.7% at 32, -26.7% at 64, -7.1% at 128,
-3.0% at 256, +0.1% at 512 (Masked, earlier unpaired run: -78% at 1 token).
Sparse combine alone gains 2.4%; the switches are not additive (section 17).
GPU suite on 24 ranks passes for every configuration tried; a collect that
waits only for channel 0 fails it; a mismatched FFN-only shape control is
rejected on every rank. Unit tests 41/41, Ruff clean. Record:
docs/validation-v1-sparse-hybrid.json.
…mains

On 200944ce, all three switches, paired against v1 (2 repeats, every
repeat faster): two FFN domains (4 FFN + 20 Attention ranks) -67.1% at 1
token to -8.6% at 128 and -1.7% at 256; three Attention domains (2 FFN + 6
Attention ranks, 3:1) -33.9% at 1 token, -35.7% at 8, -6.0% at 128. The gain
scales with the number of Attention domains. The GPU suite passes with two
FFN domains at capacities 8 and 64 on 24 ranks.
The construction-time agreement check also ran in one NVLink domain and in
mixed-role layouts, where the switches do nothing, so ranks that set them, or
the role-local AFDEP_SHARED_* controls they depend on, differently were
rejected there. get_afd_sparse_hybrid_config now reports the switches off
outside the directional hybrid (found by an independent review of in-node
invariance).

On 2d22e5e9: unit tests 41/41; GPU suite on kermit, 6 pods x 4 H200
(4 FFN + 20 Attention), all three switches, 24 ranks; the FFN-only
AFDEP_SHARED_HYBRID_FORWARD=1 still rejected there on every pod; and in one
NVLink domain (8 H200) the same control with every switch set now runs the
suite normally.

Also recorded: in-node screens (6A:2F, 7A:1F, 4A:4F, 1 to 2,048 tokens,
200944ce against 958652a3), switches off -0.8..+0.8% and on -1.2..+1.0%;
kermit single-role 5:1, all three switches -48.8% at 1 token, -50.5% at 8,
-40.8% at 32; kermit mixed-role 11:1, where the role path runs the union
hybrid and matches Masked, with 8 SMs -17% at 1 token but +56% at 512.
The channel-0 witness of the sparse combine (section 17) followed channel
0's own forward warp only. A source waits for its own channels and channel
0, so it could start its next dispatch, whose payload lands in the rail
forwarder's dispatch scale-out inbox, while other forward warps still read
the forwarder's combine scale-up inbox, which can overlap it at FP8 (found
by an independent design review). Now every forward warp releases a
rank-local DONE count after its last read, and channel 0 waits for all C
before the witness.

The GPU suite gains a one-small-source case with spread routing (a
token's experts on every FFN rank). On kermit 4F:20A (FP8, hidden 2048,
capacity 2048, 24 SMs, all three switches) the fixed tree passes with and
without forward warps delayed ~200 us. Three fault runs did not make the
unfixed tree fail, so the gap is argued from the layout and the code, not
reproduced. Cost, paired (2 repeats): +6.5% (5 us) at 8 tokens, within
1.2% at 1, 32, 128 and 512; still -47% against v1 at 8 tokens.
At high Attention:FFN ratios the FFN group is smaller than a node, so FFN
ranks share their NVLink domain with Attention ranks and the role path
falls back to the union hybrid, which matches Masked. With
AFDEP_MIXED_HYBRID=1 a layout with FFN ranks [0, F), 0 < F < U, runs the
directional protocol with one FFN-holding domain and roles per rank,
through generated copies of hybrid_dispatch.cuh and hybrid_combine.cuh:
every domain-0 GPU stays its rail's forwarder (node-wide ingress), only
ranks below F hold experts, domain-0 Attention GPUs also send (local
bypass, self counts) and reply to themselves locally, and pure-Attention
domains keep the send block and the collect. Every rank's reply inbox moves
into a slice reserved before the combine staging slice, and dispatch and
combine assert both layouts end below the slices. The agreement tuple
carries F. Shared kernels and defaults are unchanged.

kermit 6 x 4 H200, FP8, hidden 2048, top-8, capacity 2048, paired against
the v1 union role path (2 repeats, every repeat faster), on 81b71eb0:
11:1 (F=2, 128 experts) -23.2% at 1 token, -19.7% at 8, -21.7% at 32,
-14.8% at 128, -8.1% at 512; 23:1 (F=1, 64 experts) -20.7% to -6.2%.
Masked matches union. GPU suite on 24 ranks passes at F=2 (FP8 and BF16,
capacities 8 and 64), F=1 and F=3. Unit tests 41/41, Ruff clean. Not yet:
the sparse switches on this path, fault injection for the reply-slice and
staging orderings, a two-buffer interleave.
Sections 17 and 18 move under a Proposed heading. Every number names the tree it was
measured on: the 12-domain and 5:1 screens predate the done witness, which was screened
and suite-run only on kermit 4F:20A. The records map each measured tree to its
published digest, rename the control arm to v1, mark the channel-0-only witness as the
unsafe earlier tree, and drop a harness-only variable. The docs add that the switches
are read at every launch but checked once at construction, that the sparse path was run
with FP8 only, and that the mixed path was validated on 4-GPU NVLink domains only.
A section for the two opt-in additions, with three figures: the sparse switches at 11:1
over 12 domains, the sparse hybrid at other layouts, and the mixed-role path. Every
number names the tree it was measured on; the 12-domain screens predate the done
witness. The support section states the narrower validation of the proposal.
docs/v1.md is now docs/design.md, and each docs/validation-v1-<name>.json
record is docs/validation-<name>.json. Links in the README, docs and
templates, cross-references inside the records, the figure data's sources
and two comments in afdep_suites.hpp follow. Section headings are
unchanged, so every anchor still resolves. The comment edits change only
comments; LOCAL_CHANGES.json carries the refreshed file hash.
The README, design doc, notices, provenance text and records call the
role path afdEP, the role path or the release instead of v1. The design
doc's naming note becomes a digest note: 958652a3 is the tree of commit
32c871d, and the commits after it changed only comments, notices and
other text in the native tree (the 1.0.0 release tree is 131efc37). The
8 x H200 and cross-node record names its role arm afd, as the other
records do, and the role-suites record names the tree before the suites
pre_suites. Tests call the pinned shared-kernel hashes
SHARED_KERNEL_SHA256 and drop v1 from a test name.

The comment edit in buffer.hpp and the provenance text change only
comments and text; the native source digest is now db115728.
benchmarks.json names its role series afdEP, and plot_benchmarks.py
uses that label in its legends and subtitles. The regenerated result
figures differ from the previous ones only in that text and in the
legend's second entry, which moves left by the width of the dropped
" v1". The two hand-drawn diagrams change only their title, description
and heading text.
docs/v1.md becomes docs/design.md with the cross-node sections 17 and
18, and main's records take their new names. The conflicts keep the
cross-node text: the README's links to sections 17 and 18 point at
docs/design.md, docs/api.md keeps the opt-in protocols, the design
doc's digest note keeps the published digests of the cross-node trees,
and LOCAL_CHANGES.json keeps the sparse and mixed-role scope entries
after main's reworded ones.

The figure script names its role series afdEP, so the cross-node
proposal data in benchmarks.json does too, and the regenerated
results-xn figures differ only in that label, the subtitles and the
legend entries after it, which move left by the width of the dropped
" v1". The two cross-node records cite docs/design.md; their own names
and wording change in the next commit.

The merged comment edits in buffer.hpp and afdep_suites.hpp change only
comments; LOCAL_CHANGES.json carries the refreshed hashes and the
native source digest is now 233548d8.
The sparse and mixed-role records are now docs/validation-sparse-hybrid.json
and docs/validation-mixed-hybrid.json; the README, the design doc and the
mixed-role record's cross-reference follow. The arm the proposal is
paired against, the role path on 958652a3 with no sparse switches, is
the default in prose and tables: the records' v1 arms and order entries
are default, control_v1 is control_default, and the README rows say
afdEP as the figures do. The records' notes on published trees no longer
name the generation. Numbers and digests are unchanged, and the figures
regenerate identically. The native tree is unchanged (233548d8).
@Etelis
Etelis merged commit 2185d50 into main Oct 24, 2026
4 checks passed
@Etelis
Etelis deleted the cross-node branch October 24, 2026 12:26
@Etelis Etelis mentioned this pull request Oct 24, 2026
3 of 6 tasks
Etelis added a commit that referenced this pull request Oct 25, 2026
## What and why

Sections 17 to 19 (sparse and mixed-role directional hybrids, FP8
combine replies) were merged as opt-in additions in #3 and #12, but the
README and `docs/design.md` still described them as a proposal under
review on the cross-node branch.

- README: the section "Cross-node proposal (opt-in, under review)"
becomes "Cross-node additions (opt-in)". The two links to its anchor and
the numbers summary are updated to match.
- `docs/design.md`: the heading "Proposed (opt-in, under review)"
becomes "Opt-in additions after 1.0.0". It now says the sections were
merged after the 1.0.0 release (#3 and #12) and that 1.0.0 does not ship
them. The decisions in sections 17 and 19 now say "merged as opt-in, off
by default" instead of "opt-in until reviewed", "needs a maintainer
decision" and "flagged for review".

Docs only. Defaults, code and the dated validation records are
unchanged.

## Checks

- [x] `python -m pytest tests/unit`, `ruff check .` and `ruff format
--check .` pass
- [ ] Native changes: none
- [ ] Performance claims: none
- [x] Records and docs/design.md name the exact native source digest
measured (unchanged)
- [ ] Optimizations are AFD-specific and keep a control switch: not
applicable
- [x] Communication-only results are labeled as such; no serving claims
without serving evidence

## Evidence

The link check covers relative Markdown links, `src`/`srcset`, heading
anchors, and every `docs/...` path in tracked text files: 238
references, none broken.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0