8000 afdEP-comm/docs/api.md at main · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
160 lines (138 loc) · 10.2 KB

api.md

File metadata and controls

160 lines (138 loc) · 10.2 KB

API contract

Buffer(group=..., config=Config(...)) owns one NCCL communicator and one vendored DeepEP V2 ElasticBuffer. Initialization and close() are collective, outside CUDA capture. The caller owns a CPU Gloo process group, sets the current CUDA device first, and keeps the group alive until close.

Roles and tensors

FFN ranks come first; each owns num_experts / ffn_ranks consecutive real experts. Attention ranks follow. With Config(ffn_ranks_per_domain=f) the FFN ranks are instead the first f ranks of every NVLink domain (U ranks each), so every node can hold FFN ranks: FFN index j is rank (j // f) * U + j % f (Config.ffn_rank) and still owns real experts [j * E, (j + 1) * E). Buffer.role_rank is the rank's FFN or Attention index (Config.role_rank). The physical domains must match: F / f domains of W·f / F ranks, with hybrid mode on or off (direct RDMA). Every rank uses identical configuration, exchange order, bucket, and read_host_counts policy. Only one unmatched dispatch per Buffer is allowed. IDs must be unique per token, in [0, num_experts), or -1 to skip. Use buffer.topk_idx_dtype; invalid real-role routes assert on the GPU, including on graph replay. All payload tensors are on the selected CUDA device.

Operation Contract
source_dispatch(x, ids, weights, x_scale=None, *, bucket, read_host_counts=False) Attention: contiguous x [T,H], IDs/FP32 weights [T,K], T ≤ bucket ≤ capacity
worker_receive(*, bucket, read_host_counts=False) FFN: expert-major rows, weights, device expert prefix sums and opaque receipt
worker_reply(rows, receipt) FFN: contiguous BF16 [receive_rows,H], routing weight applied exactly once
source_collect(*, view=False) Attention: BF16 [T,H] summed over selected experts, in original token order

BF16 dispatch uses BF16 activations. FP8 uses float8_e4m3fn activations and FP32 block-128 scales [T,H/128]; strided scale views are supported. The library does not quantize or run GEMMs, except the opt-in lossy reply transport AFDEP_HYBRID_FP8_COMBINE below. H is a multiple of 256, K ≤ min(32,E), and real experts divide evenly over FFN ranks.

Receive prefix sum ends[e] is a device tensor: expert e occupies [align_up(ends[e-1], expert_alignment), ends[e]), with ends[-1]=0 for e=0. Padding rows are unspecified and must not be treated as live tokens. Return the entire receive shape to worker_reply, with valid expert rows in place. recv_counts_host, when requested, reports aligned per-expert counts, not raw live counts. Keep the receipt unchanged through the matching reply.

Graphs, streams and lifetime

Capture complete dispatch/reply pairs with read_host_counts=False after warming up the kernels on all ranks. Keep captured tensors, Buffer and graph storage alive through every replay. Source tensor shapes and bucket specialize the capture. Change payloads and IDs in place; encode inactive choices as -1. This is capacity-bounded graph execution, not arbitrary-shape graph replay. Graph capture/instantiation and replay are different operations; replay still has a launch cost.

read_host_counts=True performs host synchronization for exact eager receive allocation on FFN. Otherwise receive capacity is conservative and device counts identify live rows. Attention has zero receive payload rows. FFN bounds count Attention sources only. Registered transport buffers are distinct from receive tensors, except for source_collect(view=True). With one FFN rank in one NVLink domain (and no more ranks than top-k), each token's only reply row is already its final output. That call then skips the reduce kernel and returns a strided [T,H] view of the registered combine inbox. Treat the view as read-only. It is valid until the next source_dispatch on the same Buffer is enqueued; copy it if the caller needs it longer. Other layouts return the ordinary result. In one NVLink domain registered buffers are sized by role: FFN dispatch inboxes for the Attention sources and, when ranks do not exceed top-k, Attention combine inboxes for the FFN ranks; every rank registers the same size. AFDEP_UNION_BUFFERS=1 keeps DeepEP's symmetric sizing as a measurement control.

Operations order their internal communication stream with the current CUDA stream using DeepEP events. Use caller stream dependencies before passing data produced on another stream. Outputs/receipts and user inputs must remain alive until dependent GPU work completes. A completed Python pair allows the next pair to be enqueued; device-side protocol and stream ordering protect reuse. Independent Buffers do not share ordering. close() synchronizes outstanding work and frees registered storage; destroy graphs before closing the Buffer.

Several Buffers on one group

A Buffer holds one exchange at a time. To keep several in flight, as dual-batch overlap (DBO) does with one exchange per micro-batch, use Lanes:

lanes = Lanes(group=group, config=config, lanes=2)  # collective, outside capture
lanes[0].source_dispatch(...)  # each lane is an ordinary Buffer
lanes.close()

Each lane owns its communicator, registered storage, pending receipt and communication stream, so memory grows with the number of lanes. Lanes checks that every rank asks for the same number and that no two lanes share a stream. A caller that constructs the Buffers itself must set AFD_DEEPEP_PER_BUFFER_COMM_STREAM=1 (afdep.deepep.PER_BUFFER_COMM_STREAM_ENV) on every rank before the first one; Lanes sets it only while it constructs and then restores it. The guarantees below hold with separate streams.

What an integration can rely on:

  • Calls enqueue and return. With read_host_counts=False every operation enqueues GPU work ordered after the caller's current stream and returns without waiting for peers. Run each lane's calls on whichever caller stream holds its data.
  • One lane runs in call order. Dispatch, reply and the next dispatch on one Buffer execute in the order they were called, on every rank.
  • Lanes are independent. No operation on one Buffer waits for another Buffer. The caller orders lanes and adds stream dependencies when one lane's output feeds another.

Every rank must still make the same calls on the same lane in the same order. With two lanes and layers a, b, ... Attention runs dispatch 0a, dispatch 1a, collect 0a, dispatch 0b, collect 1a, dispatch 1b, and FFN runs receive 0a, receive 1a, reply 0a, receive 0b, reply 1a, receive 1b. Mapping micro-batches to lanes and choosing this order belong to the caller. The GPU suite runs this order eagerly on per-lane streams and as one captured graph (--lanes 2).

Transport and tuning

afdEP-comm keeps DeepEP's uniform logical expert numbering (Attention ranks own empty expert blocks, never legal routing targets) and its collective participation. Buffer, Config, ReceiveResult and DispatchHandle (the receipt type), all exported from afdep, are the integration API; new integrations should use them. Underneath is afdep.deepep.ElasticBuffer, which takes DeepEP's logical expert numbering and the caller's own empty and zero tensors: pass ffn_ranks=M for afdEP-comm's transport, as Buffer does from Config; add ffn_ranks_per_domain=f for a per-domain FFN prefix, whose dispatch takes real expert IDs and maps them to the logical numbering itself (the FFN blocks are no longer [0, F·E)). Without ffn_ranks it runs DeepEP V2 as upstream ships it, over a symmetric group; the benchmarks use that for the baseline, FastAFD's DeepEP M2N (benchmarks/fastafd.py).

allow_hybrid_mode, num_sms, num_qps, timeouts and RDMA traffic class pass through to the pinned transport. num_sms=0 selects approximately 24/132 of the smallest device's SM count; choose/tune explicitly for a workload. Both hybrid settings are validated in one NVLink domain and across NVLink domains joined by GIN. Across domains, role specialization runs the directional hybrid when every domain holds one role, and the union hybrid otherwise (design, sections 6 and 7). A per-domain FFN prefix puts both roles in every domain, so it runs the union hybrid's work with per-rank roles, silent FFN rails and Attention rails without the dispatch-entry barrier (section 20). Like the other hybrid controls, AFDEP_UNION_HYBRID must be set before construction. The directional hybrid always signals only the channels that carry tokens (the sparse directional hybrid, section 17 and sparse-hybrid.md); there is nothing to set. The opt-in mixed-role path (AFDEP_MIXED_HYBRID, section 18) changes the protocol when FFN ranks share a domain with Attention ranks. Set it, and any AFDEP_SHARED_* ablation control, alike on every rank before constructing the Buffer and keep them; construction gathers every rank's resolved values once and rejects a mismatch.

AFDEP_HYBRID_FP8_COMBINE=1 (section 19; validated on H200; GSM8K accuracy unchanged, no per-layer or next-token KL check yet) sends the directional hybrid's combine replies across NVLink domains as FP8: set it alike before construction, which latches it (later changes have no effect), checks it on every rank and reports it as Buffer.fp8_combine; it is off outside the directional hybrid. It is lossy: each FFN domain's BF16 partial is rounded to E4M3 with a power-of-two scale per 128 elements (error at most 2^-4 of the group's largest magnitude; tiny elements flush to zero; NaN stays in place; an infinity zeroes its group's finite elements unless they are near the BF16 range limit) before the source sums the domains in FP32. source_collect still returns BF16. AFDEP_HYBRID_FP8_COMBINE_MIN_BUCKET=b, latched and checked with it and reported as Buffer.fp8_combine_min_bucket, keeps BF16 replies for exchanges whose bucket is below b tokens (every rank passes the same bucket, so both roles agree on the row format). The default 0 means every bucket, negative values clamp to 0, and it reports 0 when FP8 is off.

Native compilation is lazy and process-locked. AFDEP_CACHE_DIR controls the extension cache; EP_JIT_CACHE_DIR controls DeepEP kernel JIT cache. Both must be writable. Use matching NCCL headers/runtime before starting Python; already loaded libraries cannot be replaced by changing environment variables later.

0