Repository navigation
Conversation
With hybrid off across NVLink domains the exit barriers stay union (system fence, flush of every QP, a signal to every rank and a wait for all), so every write of exchange N+1 still follows its reader's exit of exchange N, and the entry barriers order nothing: section 1's argument over the world GIN barrier. The global FFN prefix skips both, a per-domain prefix only combine's until its dispatch has in-kernel routes. The global prefix also drops the source finalize, and every direct layout the FFN completion kernel. The protocol is latched at construction and checked on every rank. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Every role-aware direct dispatch now runs a generated copy of dispatch.cuh with per-rank roles for a global or per-domain FFN prefix, whose notify and payload warps check each real route and map it to DeepEP's logical expert before their first route-derived index; the handle copies the logical IDs for combine. Per-domain layouts drop the separate route kernel and, like the global prefix, skip both entry barriers and the completion kernels. The protocol is unchanged. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Section 2 over GIN: sources reduce and send rank and expert counts to FFN ranks only, at the canonical positions a per-domain prefix interleaves, and FFN ranks send none and wait only for sources. Each FFN rank's counts come from one source thread, as NVLink stores to a same-node FFN rank and one bulk put to a remote one, whose doorbell also rings the aggregated rank-count put before it. The exit barriers stay union. Latched and checked like the entry-barrier skip; AFDEP_UNION_NOTIFY=1 keeps the union exchange. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Design section 21 records skipping entry barriers, in-kernel route translation and directional counts on the direct path, step by step, and which layout wins at decode sizes on one set of pods: the direct path up to 32 tokens, the hybrid from 128. R12 records the directed exit over GIN, which passed the suite and NIC-backlog fault injection but is slower than the union exit. Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
| // Copyright (c) 2026 afdEP contributors. SPDX-License-Identifier: MIT | ||
| #pragma once | ||
|
|
||
| #include <deep_ep/impls/afdep_roles.cuh> |
There was a problem hiding this comment.
Shouldn't we introduce completely new files in an afdep package? @Etelis feels a bit unserious to completely work inside the deep ep fork even for afdep only logic. Wdyt?
| } | ||
| ptx::named_barrier<kNumNotifyThreads>(kNotifyBarrierIndex); | ||
|
|
||
| // TODO: for further optimization, we can fuse rank and expert counters |
There was a problem hiding this comment.
@Etelis why leaving this for future work? We should introduce it now if it'll have perf gains since it's quite trivial imo
| gin.put<team_t>(dst_ptr, src_ptr, kNumExpertsPerRank * sizeof(int64_t), dst); | ||
| } | ||
| } | ||
| } else { |
There was a problem hiding this comment.
Can we add a test with AFDEP_UNION_NOTIFY=1 covering a per-domain FFN layout across multiple nodes? We now only test the default directional path and they have different count distribution
| // ordered by the acquire on that signal, remote ones by being posted after it. | ||
| // Every rank runs the same sequence of world barriers on one set of signal slots, | ||
| // whose targets are cumulative, so skipping requires every rank to skip, which | ||
| // construction latches and checks. A per-domain FFN prefix skips only combine's, |
There was a problem hiding this comment.
- Too many lines it's unreadable
- This comment is outdated after this new direct GIN dispatch impl (for per-domain FFN layouts,
afd_gin_directwill now selectafdep_direct_gin_dispatch.cuhand enable to skip the dispatch entry barrier)
|
@Etelis thanks! Left some comments. I think we should rethink the package structure as a whole regarding our interaction with deep ep. We start diverging too much from it to be kept as an implementation inside |
Stacked on #27 (which supersedes #22). This makes the direct path (hybrid off, one RDMA hop per token) fast for decode by porting design sections 1–3 to GIN.
What changes
dispatch.cuh). Per-node FFN layouts no longer run a separate route kernel.Results (kermit, 4-GPU nodes, FP8, hidden 4096, top-8)
Details are in
docs/design.mdsection 21.Validation
docs/validation-direct-gin-protocol.json.Not included: a directed exit over GIN (section 3). It passed the suite and NIC-backlog fault injection (both negative controls failed as they must), but it is 0.5–3.6% slower than the union exit (R12).