Repository navigation
FFN as a Service (FaaS) - #33
almogtavor wants to merge 5 commits into
Conversation
One resident launch per run picks clients like the rounds method and does the receive and the reply itself. The experts run as CUDA graphs on a second stream (persistent kernel) or inside the same launch (megakernel, FP32 without tensor cores).
The megakernel reads each 16x8 tile as 256 contiguous bytes, claims column blocks per warp without staging rows in shared memory, spreads the reply copy over all warps, and is compiled with 64 registers so four CTAs fit per SM. build_resident rejects a grid that cannot be resident at once. On the Qwen3.5-397B-A17B-FP8 layer 31 replay (ShareGPT V3) it reaches 558 exchanges/s at expert alignment 1 and 559 at 128; rounds reach 538 and 267.
| const int* state, const int* quota, | ||
| const int rotate) { | ||
| bool pending = false; | ||
| for (int i = 1; i <= num_clients; ++ i) { |
There was a problem hiding this comment.
Would one lane per client with a warp ballot make sense here, still using one CTA? I’m curious whether it would cut ready-to-receive latency with staggered arrivals or wider DP groups.
There was a problem hiding this comment.
You have one thread scanning all the rest idle iiuc.
Why not scan in parallel?
| "all_active_per_s": active / min(r["wall_s"] for r in clients), | ||
| "all_active_served": served, | ||
| "client_rates": rates, | ||
| "order_divergence": diverged / total, |
There was a problem hiding this comment.
Do you think the FFNs need a shared client order here? I’m curious what happens if one serves A then B and the other serves B then A, in both graph and resident modes.
There was a problem hiding this comment.
Two FFN workers should they do same requests or not?
| } | ||
| if (received == replied and choice != -4) | ||
| end = choice; | ||
| else if (timer() - start > timeout_ns) |
There was a problem hiding this comment.
What happens if a client arrives just as one FFN times out and another picks it up? Would the client still get all its replies?
| if mode == "push" and rank == PRODUCER: | ||
| src = [buffer.data_ptr()] * (world - 1) | ||
| dst = [peers[r].data_ptr() for r in range(world) if r != PRODUCER] | ||
| elif mode == "pull" and rank != PRODUCER: |
There was a problem hiding this comment.
What do you think about Attention pulling and reducing the same reply rows? Would you expect the FFN to move on sooner, and how would a slow reader affect Attention tail latency and Buffer reuse?
|
Remove the megakernel and fusion pls. |
|
DBO, how will we schedule different lanes? |
I added an FFN service: FFN ranks keep serving N attention clients without a fixed order.
Results, exchanges/s, 8x H100, Qwen3.5-397B-A17B-FP8 layer 31 experts replayed on ShareGPT V3 tokens (output within about 1% of the checkpoint for all three), 6 clients x 128 tokens, 2 FFN ranks:
The megakernel is the fastest setup at both alignments. At alignment 1 the margin over rounds is small (4%), because both are limited by how fast the GPU can read the expert weights (about 4.3 GB per exchange per FFN rank).
For reference on the same setup: fixed order 360 / 247, host picks 409 / 213. Mixed buffer sizes (11 ranks) were measured with rounds only, before this commit: 152 and 149.
Native source digest:
2d331b5baac3bb435564ed73b4b4c2c0eeae023ac3e014ef51db85ff87fca31a. These are communication plus expert compute numbers, not serving numbers.