8000 Reject ranks that latched different wire-visible AFDEP_* controls by almogtavor · Pull Request #21 · afdEP/afdEP-comm · GitHub
[go: up one dir, main page]

Skip to content

Reject ranks that latched different wire-visible AFDEP_* controls - #21

Open
almogtavor wants to merge 1 commit into
multi-buffer-latched-controlsfrom
protocol-controls-agreement
Open

almogtavor wants to merge 1 commit into
multi-buffer-latched-controlsfrom
protocol-controls-agreement

Conversation

@almogtavor
Copy link
Copy Markdown
Member

Resolves #7. Stacked on #20 (which latches every AFDEP_* control per Buffer at construction); merge #20 first, then this retargets to main.

Classification

Wire-visible, must agree on every rank (gathered and checked here):

Control What it changes
AFDEP_KEEP_ENTRY_BARRIERS which ranks enter the entry barriers
AFDEP_UNION_NOTIFY the count exchange
AFDEP_ALL_RANK_EXIT the exit barriers
AFDEP_GRID_SYNC_EXIT how many arrivals receivers wait for
AFDEP_UNION_BUFFERS the registered layout
AFDEP_UNION_HYBRID the hybrid protocol
AFDEP_MIXED_HYBRID the hybrid protocol and reply layout

Already checked as resolved values (unchanged): count-first (AFDEP_PAYLOAD_FIRST, AFDEP_COUNT_FIRST_MIN_BUCKET, AFDEP_COUNT_FIRST_ARM) and the sparse hybrid parts and FP8 combine replies (AFDEP_SHARED_*, AFDEP_HYBRID_FP8_COMBINE*), which #3's switches resolve into.

Rank-local launch shapes or screen knobs, allowed to differ by role: AFDEP_SHARED_ROLE_KERNELS, AFDEP_SHARED_FFN_RECEIVE, AFDEP_SHARED_HYBRID_SEND/FORWARD/COLLECT (their wire effects are covered by the resolved checks above), AFDEP_HYBRID_ROUTE_GUARD, AFDEP_ATTENTION_GRID_SYNC, AFDEP_UNION_REDUCE_PREFIX, AFDEP_COUNT_FIRST_UNGATE/UNGATED_TOKENS/WARP_COUNT/UNIT_MIN_ROWS/EPILOGUE_SMS.

Change

  • buffer.hpp: get_afd_protocol_config() reports the seven controls from the Buffer's latched snapshot.
  • afdep/deepep.py: PROTOCOL_CONTROLS; construction gathers the tuple and, on a mismatch, closes collectively and raises ValueError naming the differing controls on every rank.
  • tests/unit/test_deepep.py: mismatch test.
  • tests/gpu/protocol_agreement.py: GPU negative control; sets each control on rank 1 only, expects ValueError on every rank, then runs a correct exchange.
  • docs/api.md: the controls are measurement switches read once at construction.

Validation (CoreWeave, 4x H200, EP_DISABLE_GIN=1)

  • tests.gpu.protocol_agreement: all 7 controls rejected on all 4 ranks, no hang; the following exchange verifies. Native digest 80d292af....
  • tests.gpu.deepep bf16 and fp8: pass.
  • tests.gpu.two_buffers --env-b AFDEP_UNION_BUFFERS=1: pass.
  • Ruff clean, unit tests 49 passed.

Seven AFDEP_* controls change barriers, the count exchange, the registered
layout or the hybrid protocol: KEEP_ENTRY_BARRIERS, UNION_NOTIFY, ALL_RANK_EXIT,
GRID_SYNC_EXIT, UNION_BUFFERS, UNION_HYBRID and MIXED_HYBRID. If ranks disagree
the exchange hangs or corrupts data. The native buffer now reports them as
latched (get_afd_protocol_config), and Buffer construction gathers them and
raises ValueError naming the differing controls on every rank.

Adds a unit test and tests/gpu/protocol_agreement.py, a negative control that
sets each control on one rank only, and documents the controls in docs/api.md.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0