8000 Document and test several Buffers on one group by Etelis · Pull Request #31 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ EP_DISABLE_GIN=1 torchrun --standalone --nproc-per-node=4 \
```

Repeat with BF16, both hybrid settings, and larger capacity/hidden/top-k.
Add `--lanes 2` to also run two Buffers on the group, interleaved as DBO does.
Destructive route guards run in separate processes: `--invalid-route` accepts
`dummy`, `duplicate`, `negati 8000 ve`, or `wide`; require the route assertion in the
log. A timeout or unrelated nonzero exit is a failure, not a successful guard test.
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,8 @@ comm.close() # Collective, after all exchanges and graph replays finish.
```

Routing, quantization and expert computation belong to the caller. BF16 and
prequantized FP8 dispatch return BF16 combined outputs. See the
prequantized FP8 dispatch return BF16 combined outputs. A Buffer holds one
exchange at a time; use `Lanes` to overlap several (DBO). See the
[tensor, graph and lifecycle contract](docs/api.md).

## Support
Expand Down
11 changes: 9 additions & 2 deletions afdep/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
# SPDX-License-Identifier: Apache-2.0
"""Expert-parallel communication for Attention-FFN Disaggregation."""

from .deepep import Buffer, Config, ReceiveResult
from .deepep import Buffer, Config, DispatchHandle, Lanes, ReceiveResult

__version__ = "1.1.0"
__all__ = ["Buffer", "Config", "ReceiveResult", "__version__"]
__all__ = [
"Buffer",
"Config",
"DispatchHandle",
"Lanes",
"ReceiveResult",
"__version__",
]
61 changes: 61 additions & 0 deletions afdep/deepep.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@
from ._build import load_deepep, source_digest

if TYPE_CHECKING:
from collections.abc import Iterator

import torch
import torch.distributed as dist

Expand All @@ -31,6 +33,11 @@
HYBRID_DEFAULT_QPS = 129
DIRECT_DEFAULT_QPS = 17
INT32_MAX = (1 << 31) - 1
# Several Buffers in one process each need a communication stream of their own.
# The native runtime reads this when a Buffer is constructed. ``Lanes`` sets it
# while it constructs its Buffers; a caller constructing them itself sets it to
# "1" on every rank before the first one.
PER_BUFFER_COMM_STREAM_ENV = "AFD_DEEPEP_PER_BUFFER_COMM_STREAM"


def per_domain_layout(
Expand Down Expand Up @@ -662,6 +669,10 @@ def close(self) -> None:
class Buffer:
"""AFD roles over ElasticBuffer (FFN ranks per ``Config``), one in-flight exchange.

This is the integration API. Hold several exchanges in flight (DBO micro-batches)
with ``Lanes``, one Buffer each over the same group; see docs/api.md,
"Several Buffers on one group".

Route only real experts [0,num_experts), with -1 for unused choices and no
duplicate expert per token. Pass prequantized FP8 plus FP32 block128 scales,
or BF16. Expert rows must be weighted exactly once before worker_reply.
Expand Down Expand Up @@ -861,6 +872,56 @@ def close(self) -> None:
self._zero_output = None


class Lanes:
"""Several Buffers on one group, one in-flight exchange each.

``lanes[i]`` is an ordinary Buffer with its own communicator, registered
storage, pending receipt and communication stream, so an exchange waiting on
one lane does not hold up another. Every rank asks for the same number of
lanes and uses them in the same order; which exchange runs on which lane is
the caller's. Constructor and close are collective, outside capture.
"""

def __init__(self, *, group: dist.ProcessGroup, config: Config, lanes: int) -> None:
import torch.distributed as dist

if type(lanes) is not int or lanes < 1:
raise ValueError("lanes must be a positive int")
counts = [None] * group.size()
dist.all_gather_object(counts, lanes, group=group)
if any(count != lanes for count in counts):
raise ValueError("Every AFD rank must use the same number of lanes")
# One lane keeps the process-wide communication stream, as a lone Buffer does.
previous = os.environ.get(PER_BUFFER_COMM_STREAM_ENV)
if lanes > 1:
os.environ[PER_BUFFER_COMM_STREAM_ENV] = "1"
try:
self._buffers = tuple(
Buffer(group=group, config=config) for _ in range(lanes)
)
finally:
if previous is None:
os.environ.pop(PER_BUFFER_COMM_STREAM_ENV, None)
else:
os.environ[PER_BUFFER_COMM_STREAM_ENV] = previous
streams = [buffer.comm_stream for buffer in self._buffers]
if any(a == b for i, a in enumerate(streams) for b in streams[i + 1 :]):
raise RuntimeError("Lanes must not share a communication stream")

def __len__(self) -> int:
return len(self._buffers)

def __getitem__(self, lane: int) -> Buffer:
return self._buffers[lane]

def __iter__(self) -> Iterator[Buffer]:
return iter(self._buffers)

def close(self) -> None:
for buffer in reversed(self._buffers):
buffer.close()


# Names used by the earlier (v0) reuse connector. They alias the current implementation.
DeepEPBuffer = Buffer
DeepEPConfig = Config
8000 4 changes: 2 additions & 2 deletions benchmarks/comm.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,13 @@
import torch.distributed as dist

from afdep._build import source_digest
from afdep.deepep import PER_BUFFER_COMM_STREAM_ENV

from .fastafd import BASELINE_KEY, FastAFDBuffer, make
from .screen import DEFAULT_ARMS

METRIC = "per block: max Attention-rank CUDA-event us per exchange"
SCOPE = "communication only; init/capture excluded; graph replay included"
# Buffers coexisting in one process need separate DeepEP communication streams.
PER_BUFFER_COMM_STREAM_ENV = "AFD_DEEPEP_PER_BUFFER_COMM_STREAM"
# Arms may set AFDEP_BENCH_COLLECT_VIEW=1 (in "_env") to time source_collect(view=True).
COLLECT_VIEW = os.environ.get("AFDEP_BENCH_COLLECT_VIEW", "0") == "1"

Expand Down Expand Up @@ -183,6 +182,7 @@ def main():
"num_sms": args.num_sms,
}
if len(args.arms) > 1:
# Buffers coexisting in one process need separate communication streams.
os.environ[PER_BUFFER_COMM_STREAM_ENV] = "1"
arms = [Arm(name, overrides, common) for name, overrides in args.arms.items()]
# Roles, sources and the Attention-rank metric come from arm 0, so arms share one
Expand Down
44 changes: 42 additions & 2 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,11 +72,51 @@ pair to be enqueued; device-side protocol and stream ordering protect reuse.
Independent Buffers do not share ordering. `close()` synchronizes outstanding
work and frees registered storage; destroy graphs before closing the Buffer.

## Several Buffers on one group

A Buffer holds one exchange at a time. To keep several in flight, as dual-batch
overlap (DBO) does with one exchange per micro-batch, use `Lanes`:

```python
lanes = Lanes(group=group, config=config, lanes=2) # collective, outside capture
lanes[0].source_dispatch(...) # each lane is an ordinary Buffer
lanes.close()
```

Each lane owns its communicator, registered storage, pending receipt and
communication stream, so memory grows with the number of lanes. `Lanes` checks
that every rank asks for the same number and that no two lanes share a stream.
A caller that constructs the Buffers itself must set
`AFD_DEEPEP_PER_BUFFER_COMM_STREAM=1` (`afdep.deepep.PER_BUFFER_COMM_STREAM_ENV`)
on every rank before the first one; `Lanes` sets it only while it constructs and
then restores it. The guarantees below hold with separate streams.

What an integration can rely on:

- **Calls enqueue and return.** With `read_host_counts=False` every operation
enqueues GPU work ordered after the caller's current stream and returns without
waiting for peers. Run each lane's calls on whichever caller stream holds its data.
- **One lane runs in call order.** Dispatch, reply and the next dispatch on one Buffer
execute in the order they were called, on every rank.
- **Lanes are independent.** No operation on one Buffer waits for another Buffer.
The caller orders lanes and adds stream dependencies when one lane's output
feeds another.

Every rank must still make the same calls on the same lane in the same order.
With two lanes and layers a, b, ... Attention runs dispatch 0a, dispatch 1a,
collect 0a, dispatch 0b, collect 1a, dispatch 1b, and FFN runs receive 0a,
receive 1a, reply 0a, receive 0b, reply 1a, receive 1b. Mapping micro-batches to
lanes and choosing this order belong to the caller. The GPU suite runs this order
eagerly on per-lane streams and as one captured graph (`--lanes 2`).

## Transport and tuning

afdEP-comm keeps DeepEP's uniform logical expert numbering (Attention ranks own empty
expert blocks, never legal routing targets) and its collective participation. The
low-level connector interface is `afdep.deepep.ElasticBuffer`: pass `ffn_ranks=M` for
expert blocks, never legal routing targets) and its collective participation.
`Buffer`, `Config`, `ReceiveResult` and `DispatchHandle` (the receipt type), all
exported from `afdep`, are the integration API; new integrations should use them.
Underneath is `afdep.deepep.ElasticBuffer`, which takes DeepEP's logical expert
numbering and the caller's own empty and zero tensors: pass `ffn_ranks=M` for
afdEP-comm's transport, as `Buffer` does from `Config`; add `ffn_ranks_per_domain=f` for a
per-domain FFN prefix, whose dispatch takes real expert IDs and maps them to the logical
numbering itself (the FFN blocks are no longer `[0, F·E)`). Without `ffn_ranks` it runs DeepEP
Expand Down
Loading
Loading
0