8000 FFN as a Service (FaaS) by almogtavor · Pull Request #33 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content

FFN as a Service (FaaS) - #33

Open
almogtavor wants to merge 5 commits into
mainfrom
ffn-service
Open

almogtavor wants to merge 5 commits into
mainfrom
ffn-service

Conversation

@almogtavor
@almogtavor almogtavor commented Nov 15, 2026 •
Copy link
Copy Markdown
Member

I added an FFN service: FFN ranks keep serving N attention clients without a fixed order.

  1. Buffers of different sizes are registered once at start (tested with capacities 512, 128 and 8192 together).
  2. The FFN runs constantly. I compare three ways to do it below.
  3. The first ready client is served; when several are ready together they rotate (round-robin).
  4. Each DP group is its own client with its own buffer and is scheduled by itself.

Results, exchanges/s, 8x H100, Qwen3.5-397B-A17B-FP8 layer 31 experts replayed on ShareGPT V3 tokens (output within about 1% of the checkpoint for all three), 6 clients x 128 tokens, 2 FFN ranks:

expert alignment rounds (32 per launch) persistent kernel megakernel
1 538 394 558
128 267 210 559
  • Rounds: one CUDA graph runs 32 rounds per launch; the host only relaunches.
  • Persistent kernel: one launch for the whole run does the pick, the receive and the reply; the expert GEMMs run as graphs on a second stream. It is 0.73-0.79x of rounds: the resident CTAs spin beside the GEMMs, and their receive and reply are simpler than the DeepEP kernels.
  • Megakernel: the same launch also computes the experts, on tensor cores, with 528 CTAs (4 per SM). The weights are stored in the order the tensor cores read them. It never pads expert rows, so it gets the same number at both alignments: 1.04x of rounds at alignment 1, 2.09x at alignment 128.

The megakernel is the fastest setup at both alignments. At alignment 1 the margin over rounds is small (4%), because both are limited by how fast the GPU can read the expert weights (about 4.3 GB per exchange per FFN rank).

For reference on the same setup: fixed order 360 / 247, host picks 409 / 213. Mixed buffer sizes (11 ranks) were measured with rounds only, before this commit: 152 and 149.

Native source digest: 2d331b5baac3bb435564ed73b4b4c2c0eeae023ac3e014ef51db85ff87fca31a. These are communication plus expert compute numbers, not serving numbers.

@almogtavor
almogtavor requested a review from Etelis as a code owner November 15, 2026 14:37
@almogtavor almogtavor changed the title Add FFN service with ready-order scheduling FFN as a service Nov 15, 2026
@almogtavor almogtavor changed the title FFN as a service FFN as a Service (FaaS) Nov 15, 2026
One resident launch per run picks clients like the rounds method and does
the receive and the reply itself. The experts run as CUDA graphs on a second
stream (persistent kernel) or inside the same launch (megakernel, FP32
without tensor cores).
The megakernel reads each 16x8 tile as 256 contiguous bytes, claims column
blocks per warp without staging rows in shared memory, spreads the reply copy
over all warps, and is compiled with 64 registers so four CTAs fit per SM.
build_resident rejects a grid that cannot be resident at once.
On the Qwen3.5-397B-A17B-FP8 layer 31 replay (ShareGPT V3) it reaches 558
exchanges/s at expert alignment 1 and 559 at 128; rounds reach 538 and 267.
Comment thread afdep/service.cu
const int* state, const int* quota,
const int rotate) {
bool pending = false;
for (int i = 1; i <= num_clients; ++ i) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would one lane per client with a warp ballot make sense here, still using one CTA? I’m curious whether it would cut ready-to-receive latency with staggered arrivals or wider DP groups.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have one thread scanning all the rest idle iiuc.
Why not scan in parallel?

Comment thread tests/gpu/ffn_service.py
"all_active_per_s": active / min(r["wall_s"] for r in clients),
"all_active_served": served,
"client_rates": rates,
"order_divergence": diverged / total,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think the FFNs need a shared client order here? I’m curious what happens if one serves A then B and the other serves B then A, in both graph and resident modes.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two FFN workers should they do same requests or not?

Comment thread afdep/service.cu
}
if (received == replied and choice != -4)
end = choice;
else if (timer() - start > timeout_ns)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if a client arrives just as one FFN times out and another picks it up? Would the client still get all its replies?

Comment thread benchmarks/pull_push.py
if mode == "push" and rank == PRODUCER:
src = [buffer.data_ptr()] * (world - 1)
dst = [peers[r].data_ptr() for r in range(world) if r != PRODUCER]
elif mode == "pull" and rank != PRODUCER:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about Attention pulling and reducing the same reply rows? Would you expect the FFN to move on sooner, and how would a slow reader affect Attention tail latency and Buffer reuse?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's experimental.

@Etelis
Etelis commented Nov 16, 2026
Copy link
Copy Markdown
Member

Remove the megakernel and fusion pls.

@Etelis
Etelis commented Nov 16, 2026
Copy link
Copy Markdown
Member

DBO, how will we schedule different lanes?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0