8000 Fast decode on the direct GIN path by Etelis · Pull Request #24 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content

Fast decode on the direct GIN path - #24

Open
Etelis wants to merge 4 commits into
mainfrom
direct-gin-protocol
Open

Etelis wants to merge 4 commits into
mainfrom
direct-gin-protocol

Conversation

@Etelis
@Etelis Etelis commented Oct 27, 2026 •
Copy link
Copy Markdown
Member

Stacked on #27 (which supersedes #22). This makes the direct path (hybrid off, one RDMA hop per token) fast for decode by porting design sections 1–3 to GIN.

What changes

  • Entry barriers skipped. The exit barriers stay, so every write still follows its reader's previous exit. The setting is latched at construction and checked on every rank.
  • Routes checked and translated inside the dispatch kernel (a generated copy of dispatch.cuh). Per-node FFN layouts no longer run a separate route kernel.
  • Counts go from Attention to FFN only: NVLink stores to same-node FFN ranks, one put to remote ones.

Results (kermit, 4-GPU nodes, FP8, hidden 4096, top-8)

  • Against FastAFD's DeepEP M2N in the same per-node layouts: 21–28% faster at 1–32 tokens on 2 nodes, 18–27% on 3 nodes.
  • Same-pods comparison, 2 × 4, 6 Attention + 2 FFN:
    • Up to 32 tokens the direct path wins: 74–92 µs, against 91–119 µs for today's best (FFN on node 0, mixed-role hybrid).
    • From 128 tokens the hybrid wins.

Details are in docs/design.md section 21.

Validation

  • The GPU suite is exact after every step: 17, 18 and 18 runs on 2 × 4 and 3 × 4. That covers global and per-domain FFN prefixes, BF16 and FP8, capacities 8–512, and FastAFD.
  • Invalid routes stop at the in-kernel checks.
  • A rank with a different control setting is rejected at construction.
  • Record: docs/validation-direct-gin-protocol.json.

Not included: a directed exit over GIN (section 3). It passed the suite and NIC-backlog fault injection (both negative controls failed as they must), but it is 0.5–3.6% slower than the union exit (R12).

8000
With hybrid off across NVLink domains the exit barriers stay union
(system fence, flush of every QP, a signal to every rank and a wait for
all), so every write of exchange N+1 still follows its reader's exit of
exchange N, and the entry barriers order nothing: section 1's argument
over the world GIN barrier. The global FFN prefix skips both, a
per-domain prefix only combine's until its dispatch has in-kernel
routes. The global prefix also drops the source finalize, and every
direct layout the FFN completion kernel. The protocol is latched at
construction and checked on every rank.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Every role-aware direct dispatch now runs a generated copy of
dispatch.cuh with per-rank roles for a global or per-domain FFN prefix,
whose notify and payload warps check each real route and map it to
DeepEP's logical expert before their first route-derived index; the
handle copies the logical IDs for combine. Per-domain layouts drop the
separate route kernel and, like the global prefix, skip both entry
barriers and the completion kernels. The protocol is unchanged.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Section 2 over GIN: sources reduce and send rank and expert counts to
FFN ranks only, at the canonical positions a per-domain prefix
interleaves, and FFN ranks send none and wait only for sources. Each
FFN rank's counts come from one source thread, as NVLink stores to a
same-node FFN rank and one bulk put to a remote one, whose doorbell
also rings the aggregated rank-count put before it. The exit barriers
stay union. Latched and checked like the entry-barrier skip;
AFDEP_UNION_NOTIFY=1 keeps the union exchange.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Design section 21 records skipping entry barriers, in-kernel route
translation and directional counts on the direct path, step by step,
and which layout wins at decode sizes on one set of pods: the direct
path up to 32 tokens, the hybrid from 128. R12 records the directed
exit over GIN, which passed the suite and NIC-backlog fault injection
but is slower than the union exit.

Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
@Etelis
Etelis requested a review from almogtavor as a code owner October 27, 2026 01:43
// Copyright (c) 2026 afdEP contributors. SPDX-License-Identifier: MIT
#pragma once

#include <deep_ep/impls/afdep_roles.cuh>
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we introduce completely new files in an afdep package? @Etelis feels a bit unserious to completely work inside the deep ep fork even for afdep only logic. Wdyt?

}
ptx::named_barrier<kNumNotifyThreads>(kNotifyBarrierIndex);

// TODO: for further optimization, we can fuse rank and expert counters
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Etelis why leaving this for future work? We should introduce it now if it'll have perf gains since it's quite trivial imo

gin.put<team_t>(dst_ptr, src_ptr, kNumExpertsPerRank * sizeof(int64_t), dst);
}
}
} else {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test with AFDEP_UNION_NOTIFY=1 covering a per-domain FFN layout across multiple nodes? We now only test the default directional path and they have different count distribution

// ordered by the acquire on that signal, remote ones by being posted after it.
// Every rank runs the same sequence of world barriers on one set of signal slots,
// whose targets are cumulative, so skipping requires every rank to skip, which
// construction latches and checks. A per-domain FFN prefix skips only combine's,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Too many lines it's unreadable
  2. This comment is outdated after this new direct GIN dispatch impl (for per-domain FFN layouts, afd_gin_direct will now select afdep_direct_gin_dispatch.cuh and enable to skip the dispatch entry barrier)

@almogtavor
Copy link
Copy Markdown
Member

@Etelis thanks! Left some comments. I think we should rethink the package structure as a whole regarding our interaction with deep ep. We start diverging too much from it to be kept as an implementation inside

@Etelis
Etelis changed the base branch from ffn-domains to ffn-domains-hybrid November 1, 2026 15:41
Base automatically changed from ffn-domains-hybrid to main November 8, 2026 06:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0