8000 Support several Buffers per process with different AFDEP_* controls by almogtavor · Pull Request #20 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content

Support several Buffers per process with different AFDEP_* controls - #20

Open
almogtavor wants to merge 1 commit into
mainfrom
multi-buffer-latched-controls
Open

almogtavor wants to merge 1 commit into
mainfrom
multi-buffer-latched-controls

Conversation

@almogtavor
Copy link
Copy Markdown
Member

Resolves #5.

Problem

Native code read the AFDEP_* launch controls from the process environment on every launch (deep_ep::get_env), while each Buffer's registered layout is sized at construction. An in-process A/B harness that opens Buffer A, switches the controls and opens Buffer B leaves A running with B's launch shapes against A's layout. With AFDEP_UNION_BUFFERS=1 on B this fails every rank at the buffer.hpp:1059 buffer-size assertion; with other controls it can silently mismatch or fault.

Fix

  • csrc/utils/system.hpp: get_env reads AFDEP_* names from a thread-local snapshot when one is active (LatchedEnvScope), otherwise from the environment as before.
  • csrc/elastic/buffer.hpp: each ElasticBuffer snapshots the AFDEP_* environment at construction and activates it in the constructor, dispatch, combine and get_afd_sparse_hybrid_config.
  • LOCAL_CHANGES.json updated for both vendored files.
  • docs/api.md: several Buffers per process are supported; every rank enqueues their exchanges in one order; each Buffer keeps the controls it was opened with.
  • docs/design.md (R10): explains the old in-process A/B failure.
  • tests/gpu/two_buffers.py: opens A under the process env and B under --env-b KEY=VALUE, then checks eager alternation, one graph per Buffer replayed alternately, one graph interleaving both, and two graphs on one Buffer.

Validation (CoreWeave, 4x H200, EP_DISABLE_GIN=1, 2 attention + 2 FFN ranks)

Case main this PR
same config ok ok
--env-b AFDEP_UNION_BUFFERS=1 fails, buffer.hpp:1059 assertion on all ranks ok
--env-b AFDEP_KEEP_ENTRY_BARRIERS=1 ok ok
--env-b AFDEP_SHARED_FFN_RECEIVE=1 ok ok
--env-b AFDEP_SHARED_ROLE_KERNELS=1 ok ok
--env-b AFDEP_UNION_NOTIFY=1 ok ok

Checked exchanges per rank: eager 24, graph_alternating 96, graph_interleaved 96, two_graphs_one_buffer 96. Native source digest 4ceca478....

Also on this PR: tests.gpu.deepep bf16 and fp8 pass, Ruff clean, unit tests (48) pass.

EP_DISABLE_GIN=1 torchrun --standalone --nproc-per-node=4 -m tests.gpu.two_buffers --env-b AFDEP_UNION_BUFFERS=1

…oexist

Native code read the AFDEP_* launch controls from the process environment on
every launch, while each Buffer's registered layout is sized at construction.
A Buffer opened before an in-process A/B harness switched the controls then ran
with the other arm's launch shapes against its own layout.

Each ElasticBuffer now snapshots the AFDEP_* environment when constructed, and
its constructor, dispatch, combine and config lookup read that snapshot. Adds
tests/gpu/two_buffers.py covering same and different configs, eager and in
graphs, and documents multiple Buffers per process in docs/api.md.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two Buffers in one process: fault in the A/B harness (different launch-shape controls)

1 participant

0