Buffer(group=..., config=Config(...)) owns one NCCL communicator and one
vendored DeepEP V2 ElasticBuffer. Initialization and close() are collective,
outside CUDA capture. The caller owns a CPU Gloo process group, sets the
current CUDA device first, and keeps the group alive until close.
FFN ranks come first; each owns num_experts / ffn_ranks consecutive real experts.
Attention ranks follow. With Config(ffn_ranks_per_domain=f) the FFN ranks are instead
the first f ranks of every NVLink domain (U ranks each), so every node can hold FFN ranks:
FFN index j is rank (j // f) * U + j % f (Config.ffn_rank) and still owns real experts
[j * E, (j + 1) * E). Buffer.role_rank is the rank's FFN or Attention index
(Config.role_rank). The physical domains must match: F / f domains of W·f / F ranks,
with hybrid mode on or off (direct RDMA). Every rank uses identical configuration, exchange order,
bucket, and read_host_counts policy. Only one unmatched dispatch per Buffer
is allowed. IDs must be unique per token, in [0, num_experts), or -1 to skip.
Use buffer.topk_idx_dtype; invalid real-role routes assert on the GPU, including
on graph replay. All payload tensors are on the selected CUDA device.
| Operation | Contract |
|---|---|
source_dispatch(x, ids, weights, x_scale=None, *, bucket, read_host_counts=False) |
Attention: contiguous x [T,H], IDs/FP32 weights [T,K], T ≤ bucket ≤ capacity |
worker_receive(*, bucket, read_host_counts=False) |
FFN: expert-major rows, weights, device expert prefix sums and opaque receipt |
worker_reply(rows, receipt) |
FFN: contiguous BF16 [receive_rows,H], routing weight applied exactly once |
source_collect(*, view=False) |
Attention: BF16 [T,H] summed over selected experts, in original token order
|
BF16 dispatch uses BF16 activations. FP8 uses float8_e4m3fn activations and
FP32 block-128 scales [T,H/128]; strided scale views are supported. The library
does not quantize or run GEMMs, except the opt-in lossy reply transport
AFDEP_HYBRID_FP8_COMBINE below. H is a multiple of 256, K ≤ min(32,E), and
real experts divide evenly over FFN ranks.
Receive prefix sum ends[e] is a device tensor: expert e occupies
[align_up(ends[e-1], expert_alignment), ends[e]), with ends[-1]=0 for e=0.
Padding rows are unspecified and must not be treated as live tokens. Return the
entire receive shape to worker_reply, with valid expert rows in place.
recv_counts_host, when requested, reports aligned per-expert counts, not
raw live counts. Keep the receipt unchanged through the matching reply.
Capture complete dispatch/reply pairs with read_host_counts=False after
warming up the kernels on all ranks. Keep captured tensors, Buffer and graph
storage alive through every replay. Source tensor shapes and bucket specialize
the capture. Change payloads and IDs in place; encode inactive choices as -1.
This is capacity-bounded graph execution, not arbitrary-shape graph replay.
Graph capture/instantiation and replay are different operations; replay still
has a launch cost.
read_host_counts=True performs host synchronization for exact eager receive
allocation on FFN. Otherwise receive capacity is conservative and device counts
identify live rows. Attention has zero receive payload rows. FFN bounds count
Attention sources only. Registered transport buffers are distinct from receive
tensors, except for source_collect(view=True). With one FFN rank in one NVLink
domain (and no more ranks than top-k), each token's only reply row is already its
final output. That call then skips the reduce kernel and returns a strided [T,H]
view of the registered combine inbox. Treat the view as read-only. It is valid
until the next source_dispatch on the same Buffer is enqueued; copy it if the
caller needs it longer. Other layouts return the ordinary result. In one NVLink domain
registered buffers are sized by role: FFN dispatch inboxes for the Attention sources and,
when ranks do not exceed top-k, Attention combine inboxes for the FFN ranks; every rank
registers the same size. AFDEP_UNION_BUFFERS=1 keeps DeepEP's symmetric sizing as a
measurement control.
Operations order their internal communication stream with the current CUDA
stream using DeepEP events. Use caller stream dependencies before passing data
produced on another stream. Outputs/receipts and user inputs must remain alive
until dependent GPU work completes. A completed Python pair allows the next
pair to be enqueued; device-side protocol and stream ordering protect reuse.
Independent Buffers do not share ordering. close() synchronizes outstanding
work and frees registered storage; destroy graphs before closing the Buffer.
A Buffer holds one exchange at a time. To keep several in flight, as dual-batch
overlap (DBO) does with one exchange per micro-batch, use Lanes:
lanes = Lanes(group=group, config=config, lanes=2) # collective, outside capture
lanes[0].source_dispatch(...) # each lane is an ordinary Buffer
lanes.close()Each lane owns its communicator, registered storage, pending receipt and
communication stream, so memory grows with the number of lanes. Lanes checks
that every rank asks for the same number and that no two lanes share a stream.
A caller that constructs the Buffers itself must set
AFD_DEEPEP_PER_BUFFER_COMM_STREAM=1 (afdep.deepep.PER_BUFFER_COMM_STREAM_ENV)
on every rank before the first one; Lanes sets it only while it constructs and
then restores it. The guarantees below hold with separate streams.
What an integration can rely on:
- Calls enqueue and return. With
read_host_counts=Falseevery operation enqueues GPU work ordered after the caller's current stream and returns without waiting for peers. Run each lane's calls on whichever caller stream holds its data. - One lane runs in call order. Dispatch, reply and the next dispatch on one Buffer execute in the order they were called, on every rank.
- Lanes are independent. No operation on one Buffer waits for another Buffer. The caller orders lanes and adds stream dependencies when one lane's output feeds another.
Every rank must still make the same calls on the same lane in the same order.
With two lanes and layers a, b, ... Attention runs dispatch 0a, dispatch 1a,
collect 0a, dispatch 0b, collect 1a, dispatch 1b, and FFN runs receive 0a,
receive 1a, reply 0a, receive 0b, reply 1a, receive 1b. Mapping micro-batches to
lanes and choosing this order belong to the caller. The GPU suite runs this order
eagerly on per-lane streams and as one captured graph (--lanes 2).
afdEP-comm keeps DeepEP's uniform logical expert numbering (Attention ranks own empty
expert blocks, never legal routing targets) and its collective participation.
Buffer, Config, ReceiveResult and DispatchHandle (the receipt type), all
exported from afdep, are the integration API; new integrations should use them.
Underneath is afdep.deepep.ElasticBuffer, which takes DeepEP's logical expert
numbering and the caller's own empty and zero tensors: pass ffn_ranks=M for
afdEP-comm's transport, as Buffer does from Config; add ffn_ranks_per_domain=f for a
per-domain FFN prefix, whose dispatch takes real expert IDs and maps them to the logical
numbering itself (the FFN blocks are no longer [0, F·E)). Without ffn_ranks it runs DeepEP
V2 as upstream ships it, over a symmetric group; the benchmarks use that for the
baseline, FastAFD's DeepEP M2N (benchmarks/fastafd.py).
allow_hybrid_mode, num_sms, num_qps, timeouts and RDMA traffic class pass
through to the pinned transport. num_sms=0 selects approximately 24/132 of the
smallest device's SM count; choose/tune explicitly for a workload. Both hybrid
settings are validated in one NVLink domain and across NVLink domains joined by GIN.
Across domains, role specialization runs the directional hybrid when every domain
holds one role, and the union hybrid otherwise (design, sections 6 and 7).
A per-domain FFN prefix puts both roles in every domain, so it runs the union hybrid's work
with per-rank roles, silent FFN rails and Attention rails without the dispatch-entry barrier
(section 20).
Like the other hybrid controls, AFDEP_UNION_HYBRID must be set before construction.
The directional hybrid always signals only the channels that carry tokens (the sparse
directional hybrid, section 17 and sparse-hybrid.md); there is
nothing to set. The opt-in mixed-role path (AFDEP_MIXED_HYBRID, section 18) changes
the protocol when FFN ranks share a domain with Attention ranks. Set it, and any
AFDEP_SHARED_* ablation control, alike on every rank before constructing the Buffer
and keep them; construction gathers every rank's resolved values once and rejects a
mismatch.
AFDEP_HYBRID_FP8_COMBINE=1 (section 19; validated on H200; GSM8K accuracy unchanged, no per-layer or next-token KL check yet) sends the
directional hybrid's combine replies across NVLink domains as FP8: set it alike before
construction, which latches it (later changes have no effect), checks it on every rank and
reports it as Buffer.fp8_combine; it is off outside the directional hybrid. It is lossy:
each FFN domain's BF16 partial is rounded to E4M3 with a power-of-two scale per 128 elements
(error at most 2^-4 of the group's largest magnitude; tiny elements flush to zero; NaN stays
in place; an infinity zeroes its group's finite elements unless they are near the BF16 range
limit) before the source sums the domains in FP32. source_collect still returns BF16.
AFDEP_HYBRID_FP8_COMBINE_MIN_BUCKET=b, latched and checked with it and reported as
Buffer.fp8_combine_min_bucket, keeps BF16 replies for exchanges whose bucket is below b
tokens (every rank passes the same bucket, so both roles agree on the row format). The
default 0 means every bucket, negative values clamp to 0, and it reports 0 when FP8 is off.
Native compilation is lazy and process-locked. AFDEP_CACHE_DIR controls the
extension cache; EP_JIT_CACHE_DIR controls DeepEP kernel JIT cache. Both must be
writable. Use matching NCCL headers/runtime before starting Python; already
loaded libraries cannot be replaced by changing environment variables later.