8000 Cross-node AFD: sparse and mixed-role directional hybrids (opt-in) by Etelis · Pull Request #3 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ body:
description: >-
For optimizations: which property of the Attention and FFN roles (one-way dataflow,
idle role resources, the Attention:FFN ratio) makes it AFD-specific. A change that
would speed up Masked DeepEP as well belongs upstream; see docs/v1.md.
would speed up Masked DeepEP as well belongs upstream; see docs/design.md.
- type: textarea
id: evidence
attributes:
Expand Down
4 changes: 2 additions & 2 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
- [ ] `python -m pytest tests/unit`, `ruff check .` and `ruff format --check .` pass
- [ ] Native changes: the GPU suite on the affected backend (BF16 and FP8, eager and graphs, empty and skewed routes, delayed peers, buffer reuse)
- [ ] Performance claims: paired fresh-process screens against Masked DeepEP and the change's own control switch, with each arm's native source digest
- [ ] Records and docs/v1.md name the exact native source digest measured
- [ ] Optimizations are AFD-specific and keep a control switch (docs/v1.md)
- [ ] Records and docs/design.md name the exact native source digest measured
- [ ] Optimizations are AFD-specific and keep a control switch (docs/design.md)
- [ ] Communication-only results are labeled as such; no serving claims without serving evidence

## Evidence
308 changes: 246 additions & 62 deletions README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion THIRD_PARTY_NOTICES.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ and its afdEP role specializations are MIT-licensed.

The snapshot includes AFD plugin patches for optional GIN and module-local
bindings. afdEP v0 changes role-specific receive bounds, empty-side epilogues,
route validation, and NCCL version diagnostics. afdEP v1 also changes the
route validation, and NCCL version diagnostics. afdEP also changes the
role path's synchronization and protocol: it skips the entry barriers, sends
counts from Attention to FFN only, replaces the exit barriers with directed
per-CTA arrival, sizes receive buffers per role, adds a directional hybrid
Expand Down
8 changes: 8 additions & 0 deletions afdep/deepep.py
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,14 @@ def __init__(
if count_first[0][1] not in ("full", "source"):
self.close()
raise ValueError("AFDEP_COUNT_FIRST_ARM must be full or source")
# So do the sparse directional hybrid switches, as each rank resolves them
# (role-local shape controls can turn them off on one role only).
sparse = self._gather(tuple(self._runtime.get_afd_sparse_hybrid_config()))
if any(value != sparse[0] for value in sparse):
self.close()
raise ValueError(
"Every DeepEP rank must use the same sparse hybrid configuration"
)

def _gather(self, value) -> list:
values = [None] * self._group.size()
Expand Down
63 changes: 43 additions & 20 deletions afdep/third_party/deepep_v2/LOCAL_CHANGES.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion afdep/third_party/deepep_v2/PROVENANCE.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"intermediary_commit": "3c7161949310b6d59d6b4cf9bf997a4935c8113b",
"pin_limit": "The original DeepEP revision is not recorded in the FastAFD snapshot; the immutable plugin revision and complete file hashes identify this vendored input.",
"local_changes": [
"Opt-in afdEP role specialization: receive allocation, fused direct-dispatch route checks and empty-side completion through normal stream events (v0); entry-barrier removal, directional counts, directed per-CTA exit, role-sized buffers, the directional hybrid protocol and generated role kernels (v1). Exact added and modified hashes in LOCAL_CHANGES.json. Frozen file_sha256 identifies the unchanged origin snapshot."
"Opt-in afdEP role specialization: receive allocation, fused direct-dispatch route checks and empty-side completion through normal stream events (v0), followed by entry-barrier removal, directional counts, directed per-CTA exit, role-sized buffers, the directional hybrid protocol and generated role kernels. Exact added and modified hashes in LOCAL_CHANGES.json. Frozen file_sha256 identifies the unchanged origin snapshot."
],
"inherited_changes": [
"AFD_DEEPEP_PER_BUFFER_COMM_STREAM environment variable naming.",
Expand Down
93 changes: 84 additions & 9 deletions afdep/third_party/deepep_v2/csrc/elastic/buffer.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ class ElasticBuffer {
layout::WorkspaceLayout::get_num_bytes() + num_buffer_bytes, kBufferAlignment,
allow_hybrid_mode, sl_idx, num_allocated_qps);

// afdEP count-first: the role path of one NVLink domain with every v1 role protocol;
// afdEP count-first: the role path of one NVLink domain with every role protocol;
// afdep/deepep.py checks that every rank latched the same configuration and a valid arm.
// Nothing is checked here: one rank throwing would leave its peers in the collective
// window registration above. Generic buffers never read the screen arm.
Expand Down Expand Up @@ -194,6 +194,24 @@ class ElasticBuffer {
return {afd_cf_min_bucket, afd_cf_arm};
}

// afdEP sparse directional hybrid (afdep_suites.hpp): (stagger bound, sparse dispatch, sparse
// combine) as this rank resolves them, which must agree on every rank. Only the directional
// hybrid reads them, so any other topology (one NVLink domain, mixed-role domains) reports
// them off and never rejects ranks that set them differently.
// The mixed-role hybrid (afdep_roles.hpp) is reported the same way: its F, wire-visible.
std::tuple<int, bool, bool, int> get_afd_sparse_hybrid_config() const {
if (afd_ffn_ranks > 0) {
const int mixed = afdep_mixed_hybrid(afd_ffn_ranks, nccl_context->num_scaleout_ranks,
nccl_context->num_scaleup_ranks);
if (mixed > 0)
return {0, false, false, mixed};
}
if (afd_ffn_ranks <= 0 or afdep_directional_hybrid(afd_ffn_ranks, nccl_context->num_scaleout_ranks,
nccl_context->num_scaleup_ranks) == 0)
return {0, false, false, 0};
return {afdep_hybrid_stagger_max_tokens(), afdep_hybrid_sparse_dispatch(), afdep_hybrid_sparse_combine(), 0};
}

// ReSharper disable once CppMemberFunctionMayBeStatic
void barrier(const bool& use_comm_stream, const bool& with_cpu_sync) const {
const auto compute_stream = at::cuda::getCurrentCUDAStream();
Expand Down Expand Up @@ -640,6 +658,16 @@ class ElasticBuffer {
num_scaleout_ranks * (num_max_tokens_per_rank + kNumMaxChannels)).get_num_bytes();
}

// afdEP mixed-role hybrid: every rank's combine reply inbox, in a slice reserved just before the
// staging slice (regions as the kernel's get_num_tokens_in_layout for the scale-out layout).
static int64_t get_hybrid_combine_reply_bytes(const int& num_max_tokens_per_rank, const int& hidden, const int& num_topk,
const int& num_scaleout_ranks, const bool& allow_multiple_reduction) {
const auto token_layout = get_combine_token_layout(hidden, sizeof(nv_bfloat16), num_topk);
return layout::BufferLayout<false>(
token_layout, allow_multiple_reduction ? std::min(num_scaleout_ranks, num_topk) : num_topk,
num_max_tokens_per_rank).get_num_bytes();
}

// afdEP single-FFN collect view: one FFN rank, one NVLink domain, rank layout.
bool afd_single_ffn_collect_rows(const bool& afd_direct_nvlink, const int& num_topk) const {
return afd_direct_nvlink and afd_ffn_ranks == 1 and allow_multiple_reduction and
Expand Down Expand Up @@ -681,6 +709,16 @@ class ElasticBuffer {
num_scaleout_ranks, num_scaleup_ranks,
is_scaleup_nvlink, allow_multiple_reduction, afd_role_ffn_ranks);

// afdEP mixed-role hybrid (afdep_roles.hpp): the reply inbox and the combine staging of every
// rank get slices past both layouts.
if (afd_ffn_ranks > 0 and afdep_mixed_hybrid(afd_ffn_ranks, num_scaleout_ranks, num_scaleup_ranks) > 0) {
return std::max(num_dispatch_bytes, num_combine_bytes) +
get_hybrid_combine_reply_bytes(num_max_tokens_per_rank, hidden, num_topk, num_scaleout_ranks,
allow_multiple_reduction) +
get_hybrid_combine_send_bytes(num_max_tokens_per_rank, hidden, num_topk, num_scaleout_ranks,
allow_multiple_reduction);
}

// afdEP directional hybrid with two or more Attention domains: FFN combine staging must not
// overlap any inbox another role or domain writes (the dispatch inboxes), so it gets its
// own slice past both layouts.
Expand Down Expand Up @@ -816,7 +854,10 @@ class ElasticBuffer {
// All new tensor allocations should happen after this
const auto compute_stream = stream_control_prologue(previous_event, allocate_on_comm_stream, async_with_compute_stream);

const int afd_ffn_scaleout_ranks = afd_role_aware and not deterministic ?
// afdEP mixed-role hybrid (afdep_roles.hpp): one FFN-holding domain shared with Attention ranks.
const int afd_mixed_ffn_ranks = afd_role_aware and not deterministic ?
afdep_mixed_hybrid(afd_ffn_ranks, nccl_context->num_scaleout_ranks, nccl_context->num_scaleup_ranks) : 0;
const int afd_ffn_scaleout_ranks = afd_mixed_ffn_ranks > 0 ? 1 : afd_role_aware and not deterministic ?
afdep_directional_hybrid(afd_ffn_ranks, nccl_context->num_scaleout_ranks,
nccl_context->num_scaleup_ranks) : 0;
const bool afd_hybrid_routes_in_kernel = afd_ffn_scaleout_ranks > 0 and afdep_hybrid_routes_in_kernel();
Expand Down Expand Up @@ -998,6 +1039,20 @@ class ElasticBuffer {
num_max_tokens_per_rank, hidden, num_sf_packs, num_topk, x.element_size(),
nccl_context->num_scaleout_ranks, nccl_context->num_scaleup_ranks,
nccl_context->is_scaleup_nvlink, afd_role_buffers ? afd_ffn_ranks : 0) <= num_buffer_bytes);
// afdEP mixed-role hybrid: this call's dispatch layout ends below the reply and staging slices
// its combine places at the window end (same bucket and top-k).
if (afd_ffn_ranks > 0 and afdep_mixed_hybrid(afd_ffn_ranks, nccl_context->num_scaleout_ranks,
nccl_context->num_scaleup_ranks) > 0) {
EP_HOST_ASSERT(get_dispatch_buffer_size(
num_max_tokens_per_rank, hidden, num_sf_packs, num_topk, x.element_size(),
nccl_context->num_scaleout_ranks, nccl_context->num_scaleup_ranks,
nccl_context->is_scaleup_nvlink, 0) +
get_hybrid_combine_reply_bytes(num_max_tokens_per_rank, hidden, num_topk,
nccl_context->num_scaleout_ranks, allow_multiple_reduction) +
get_hybrid_combine_send_bytes(num_max_tokens_per_rank, hidden, num_topk,
nccl_context->num_scaleout_ranks, allow_multiple_reduction)
<= num_buffer_bytes);
}

// afdEP count-first (afdep_suites.hpp): one decision per exchange from the constructor latch
// and the bucket, which every rank passes identically. The environment is re-read on every
Expand Down Expand Up @@ -1039,9 +1094,11 @@ class ElasticBuffer {
else if (afd_direct_nvlink and not afd_source and afdep_skip_entry_barriers() and afdep_directional_notify() and
afdep_directed_exit() and afdep_ffn_receive_kernels())
afd_dispatch_shape = AfdLaunchShape::kRoleKernel;
else if (afd_ffn_scaleout_ranks > 0 and afd_source and afdep_hybrid_send_blocks())
else if (afd_ffn_scaleout_ranks > 0 and afd_source and afdep_hybrid_send_blocks() and
(afd_mixed_ffn_ranks == 0 or nccl_context->scaleout_rank_idx >= afd_ffn_scaleout_ranks))
afd_dispatch_shape = AfdLaunchShape::kSendBlock;
else if (afd_ffn_scaleout_ranks > 0 and not afd_source and afdep_hybrid_forward_kernels())
else if (afd_ffn_scaleout_ranks > 0 and not afd_source and afdep_hybrid_forward_kernels() and
afd_mixed_ffn_ranks == 0)
afd_dispatch_shape = AfdLaunchShape::kRoleKernel;
launch_dispatch(x.data_ptr(), sf_ptr,
topk_idx.data_ptr<topk_idx_t>(), topk_weights_ptr,
Expand Down Expand Up @@ -1076,7 +1133,8 @@ class ElasticBuffer {
afd_source and afd_single_ffn_collect_rows(afd_direct_nvlink, num_topk) ?
get_combine_token_layout(hidden, sizeof(nv_bfloat16), num_topk).get_num_bytes<false>() : 0,
afd_dispatch_shape,
afdep_count_first_ungated_tokens(), afdep_count_first_thread_count());
afdep_count_first_ungated_tokens(), afdep_count_first_thread_count(),
afd_mixed_ffn_ranks);

// Received token counters
int num_recv_tokens = 0, num_expanded_tokens = 0;
Expand Down Expand Up @@ -1391,11 +1449,25 @@ class ElasticBuffer {
const int transport_reduced_tokens = afd_ffn_ranks > 0 ?
num_max_tokens_per_rank * nccl_context->num_ranks : num_reduced_tokens;
// afdEP directional hybrid: FFN combine staging uses the slice reserved at the window end.
const int afd_ffn_scaleout_ranks = afd_ffn_ranks > 0 and not deterministic ?
const int afd_mixed_ffn_ranks = afd_ffn_ranks > 0 and not deterministic ?
afdep_mixed_hybrid(afd_ffn_ranks, nccl_context->num_scaleout_ranks, nccl_context->num_scaleup_ranks) : 0;
const int afd_ffn_scaleout_ranks = afd_mixed_ffn_ranks > 0 ? 1 : afd_ffn_ranks > 0 and not deterministic ?
afdep_directional_hybrid(afd_ffn_ranks, nccl_context->num_scaleout_ranks,
nccl_context->num_scaleup_ranks) : 0;
void* afd_scaleout_send_base = nullptr;
if (afd_ffn_scaleout_ranks > 0 and nccl_context->num_scaleout_ranks - afd_ffn_scaleout_ranks >= 2) {
if (afd_mixed_ffn_ranks > 0) {
// afdEP mixed-role hybrid: staging slice at the window end, every rank's reply inbox before it,
// both past this call's combine layout.
const auto num_send_bytes = get_hybrid_combine_send_bytes(
num_max_tokens_per_rank, hidden, num_topk, nccl_context->num_scaleout_ranks, allow_multiple_reduction);
const auto num_reply_bytes = get_hybrid_combine_reply_bytes(
num_max_tokens_per_rank, hidden, num_topk, nccl_context->num_scaleout_ranks, allow_multiple_reduction);
EP_HOST_ASSERT(get_combine_buffer_size(num_max_tokens_per_rank, hidden, num_topk,
nccl_context->num_scaleout_ranks, nccl_context->num_scaleup_ranks,
nccl_context->is_scaleup_nvlink, allow_multiple_reduction, 0) +
num_reply_bytes + num_send_bytes <= num_buffer_bytes);
afd_scaleout_send_base = math::advance_ptr(buffer, num_buffer_bytes - num_send_bytes);
} else if (afd_ffn_scaleout_ranks > 0 and nccl_context->num_scaleout_ranks - afd_ffn_scaleout_ranks >= 2) {
const auto num_send_bytes = get_hybrid_combine_send_bytes(
num_max_tokens_per_rank, hidden, num_topk, nccl_context->num_scaleout_ranks, allow_multiple_reduction);
// calculate_buffer_size reserved this slice past both layouts (sized for top-k 32).
Expand All @@ -1405,7 +1477,8 @@ class ElasticBuffer {
// afdEP role suites (afdep_suites.hpp): in directional hybrid the Attention receive runs
// the one-CTA collect kernel.
auto afd_combine_shape = AfdLaunchShape::kShared;
if (afd_ffn_scaleout_ranks > 0 and not afd_worker and afdep_hybrid_collect_kernels())
if (afd_ffn_scaleout_ranks > 0 and not afd_worker and afdep_hybrid_collect_kernels() and
(afd_mixed_ffn_ranks == 0 or nccl_context->scaleout_rank_idx >= afd_ffn_scaleout_ranks))
afd_combine_shape = AfdLaunchShape::kRoleKernel;
const auto reduce_buffer = launch_combine(
x.data_ptr(),
Expand All @@ -1432,7 +1505,8 @@ class ElasticBuffer {
afd_ffn_scaleout_ranks,
afd_direct and afd_ffn_ranks == 1 and afdep_single_ffn_reduce(),
afd_scaleout_send_base,
afd_combine_shape);
afd_combine_shape,
afd_mixed_ffn_ranks);

// Allocate output tensors
// afdEP single-FFN collect view: with one FFN rank in one NVLink domain (rank layout), each
Expand Down Expand Up @@ -1504,6 +1578,7 @@ static void register_apis(pybind11::module_& m) {
.def("get_physical_domain_size", &ElasticBuffer::get_physical_domain_size)
.def("get_logical_domain_size", &ElasticBuffer::get_logical_domain_size)
.def("get_afd_count_first_config", &ElasticBuffer::get_afd_count_first_config)
.def("get_afd_sparse_hybrid_config", &ElasticBuffer::get_afd_sparse_hybrid_config)
.def("barrier", &ElasticBuffer::barrier)
.def("engram_write", &ElasticBuffer::engram_write)
.def("engram_fetch", &ElasticBuffer::engram_fetch)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,34 @@ static int afdep_directional_hybrid(const int afd_ffn_ranks, const int num_scale
return afd_ffn_ranks / num_scaleup_ranks;
}

// Mixed-role directional hybrid (opt-in, AFDEP_MIXED_HYBRID=1): FFN ranks [0, F) with
// 0 < F < U fill part of NVLink domain 0, which they share with U - F Attention ranks
// (A0), and every other domain holds only Attention ranks (AP). The directional
// protocol above applies with one FFN-holding domain and roles per rank: every domain-0
// GPU stays its rail's forwarder, so remote tokens still enter through all U NICs; only
// ranks below F hold experts; A0 GPUs are also sources, whose own tokens take the local
// bypass into their own forwarder (rail lane 0) and whose replies their own forward
// warps write locally. Its orderings are the directional ones plus:
// - Domain 0 keeps all three scale-up barriers (dispatch entry and exit, combine entry)
// across all U GPUs, A0 included, so state shared inside domain 0 (scale-up inboxes,
// tails, counts, list ends) is ordered as in union.
// - An AP writes dispatch N+1 into forwarder (0,u) only after its collect N saw (0,u)'s
// signal on every channel, each sent after that channel's combine-N reads; the one
// region (0,u) still reads afterwards is A0's reply inbox (its reduce epilogue), which
// therefore moves, on every rank, into a slice reserved before the combine staging
// slice at the window end, past both layouts (calculate_buffer_size).
// - Forwarders stage AP replies in the section-11 staging slice with two or more AP
// domains; A0 never stages dispatch tokens (all are local) and nothing is written into
// an AP window except its reply slice and signal words.
// Returns F, or 0 (union, or the directional protocol when F % U == 0).
static int afdep_mixed_hybrid(const int afd_ffn_ranks, const int num_scaleout_ranks,
const int num_scaleup_ranks) {
if (afd_ffn_ranks <= 0 or num_scaleout_ranks <= 1 or afd_ffn_ranks >= num_scaleup_ranks or
deep_ep::get_env<int>("AFDEP_UNION_HYBRID", 0) != 0 or deep_ep::get_env<int>("AFDEP_MIXED_HYBRID", 0) == 0)
return 0;
return afd_ffn_ranks;
}

// Directional hybrid dispatch, and direct dispatch over GIN (hybrid off across
// NVLink domains), validate each route in their notify and payload warps before
// the first route-derived index, store or put, as the direct NVLink path does,
Expand Down
Loading
Loading
0