Repository navigation
Releases
afdEP-comm v1.1.0
afdEP-comm 1.1.0: expert-parallel communication for Attention–FFN Disaggregation. Every result is measured against FastAFD's DeepEP M2N, the way FastAFD and the AFD plugin run AFD over DeepEP V2. Design and per-change evidence are in docs/design.md.
FFN ranks on every node (Config(ffn_ranks_per_domain=f), section 20): the first f GPUs of every node are FFN ranks, so a deployment can grow by adding identical nodes, such as 2 × 4A1F. Callers keep real expert IDs. FFN GPUs skip all cross-node work, and Attention GPUs no longer wait at a cross-node barrier before each dispatch. Communication only on 2 × 4 and 3 × 4 H200 (FP8), this layout is 1.4–8.5% faster than before these changes and 3.4–16.1% faster than FastAFD's DeepEP M2N in the same layout. Putting the FFN on the first nodes is still faster below 512 tokens. With hybrid mode off it runs one RDMA hop per token. Nodes with different GPU counts are not supported.
Sparse directional hybrid, now the default when every NVLink domain holds one role (section 17): only the channels that carry tokens are signaled. One token at 11:1 over 12 domains takes 74 µs instead of 231 µs, 78% below FastAFD's DeepEP M2N. In serving (Qwen3-235B, 20A:4F) it cut median TPOT by 25% at 4 requests per Attention GPU and 10% at 64.
Opt-in across NVLink domains:
- Mixed-role directional hybrid (
AFDEP_MIXED_HYBRID=1, prototype, section 18), for FFN ranks that fill only part of a node: 23% lower at one token and 8% at 512 tokens at 11:1 than the default union hybrid. Validated on 4-GPU NVLink domains only. - FP8 combine replies (
AFDEP_HYBRID_FP8_COMBINE=1, section 19): reduced replies cross NVLink domains as E4M3 with one scale per 128 elements, 13–21% lower at 512 tokens with two or more Attention domains. It is lossy; GSM8K on Qwen3-235B matched BF16 within run-to-run noise.
Fix: DeepEP V2's direct (non-hybrid) GIN barrier let a same-node receiver read NVLink rows before they arrived when a node holds both senders and receiv 9156 ers. It now fences at system scope before signaling. FastAFD's DeepEP M2N has the same bug.
Breaking changes:
- The pip package is now
afdep-comm; the import is stillafdep. Config.role_awareis removed, andBufferalways runs afdEP-comm's transport. The low-levelafdep.deepep.ElasticBufferruns it withffn_ranks=M, and runs DeepEP V2 as upstream ships it without.
Support: Hopper (SM90), BF16 and FP8, eager and CUDA graphs, one NVLink domain or several joined by GIN. Validated with PyTorch 2.13.0+cu130, CUDA 13.0 and NCCL 2.30.7; the layouts each path passed the GPU suite on are listed under Support. Blackwell (SM100) is tracked in #2.
Full changelog: v1.0.0...v1.1.0
afdEP v1.0.0
afdEP v1: expert-parallel communication specialized for Attention–FFN Disaggregation.
v1 vendors DeepEP V2's kernels and removes what the Attention and FFN roles make unnecessary, against the same kernels run without roles (Masked DeepEP). Every change keeps a control switch; details and per-change evidence are in docs/v1.md.
In one NVLink domain: no entry barriers, counts from Attention to FFN only, directed exit (only receivers wait), receive buffers sized per role, role suites that launch only the warps each role uses, and a single-FFN reduction and collect view.
Across NVLink domains (NCCL GIN): a directional protocol when every domain holds one role. Domains that hold both roles run DeepEP's union hybrid.
Measured (communication only, round trip under CUDA graphs, against Masked DeepEP; paired geometric-mean changes over fresh-process repeats):
- 4 × H100, one NVLink domain (tree
bd4db4da, sections 1–4): 27–35% lower latency at one token and 1–5% at 2,048 tokens, faster in every repeat. - 8 × H200, one NVLink domain: 22% (4A:4F), 24% (6A:2F) and 39% (7A:1F) lower at one token.
- H200 across NVLink domains with one role per domain: 30% lower at 11:1 over 12 domains and 26% at 5:1 over 6 domains at one token. Mixed-role domains match Masked within 1.4%.
Support: Hopper (SM90), BF16 and FP8, eager and CUDA graphs, one NVLink domain or several joined by GIN. The release tree passes the GPU suite in one 4-GPU NVLink domain, and the directional protocol passed it on earlier trees over two to five NVLink domains. The 8-GPU-domain and 24-rank results are measured, not suite-validated. Blackwell (SM100) is tracked in #2.
Naming: this generation was developed as v2.1; the earlier DeepEP-reuse generation is now v0. Records cite the native source digests of the trees as measured. 958652a3 is v1 before the rename; the rename and the release's comment fixes changed only comments and text, giving 131efc37 for this release.