Repository navigation
feat(ess): implement resumption stream consumer and worker - #890
Merged
Merged
Conversation
Implement background consumer loop in Experiment State Server (ESS) listening to `xmc-scheduling-resume-stream` via consumer group `ess-group`. The consumer parses `batch_size` and `max_backlog_threshold` from the trigger payload, dispatches execution to a stub resumption worker, and is activated strictly behind the `PauseSchedulingEnabled` feature flag. Encapsulate stream consumer lifecycle in `resumption.Consumer` with bounded graceful drain timeout and deadlock-free teardown hooks registered in the server cleanup stack. TAG=agy CONV=b35410bc-068d-4e00-bae0-22b3ca180c39
There was a problem hiding this comment.
Code Review
This pull request introduces a new resumption package to coordinate background consumption of the scheduling resumption stream from Redis when the PauseSchedulingEnabled feature flag is active. It includes a consumer, a task handler, payload parsing with defaults, a stub worker, and comprehensive unit tests. The review feedback highlights two key improvements: adding defensive nil checks for the handler and its worker in HandleTask to prevent potential runtime panics, and fixing a goroutine leak in TestConsumer_DrainTimeout by ensuring the simulated in-flight worker's WaitGroup is properly decremented.
8000
Add defensive nil checks for handler and worker in HandleTask to prevent runtime panics if invoked on a nil receiver or uninitialized worker. Add defer consumer.wg.Done() in TestConsumer_DrainTimeout to ensure the background wait goroutine unblocks and exits cleanly after test completion. TAG=agy CONV=b35410bc-068d-4e00-bae0-22b3ca180c39
Rename StubWorker to ResumptionWorker and NewStubWorker to NewResumptionWorker to establish the final domain worker type ahead of the downstream Spanner replay worker implementation (Task 15a). TAG=agy CONV=b35410bc-068d-4e00-bae0-22b3ca180c39
andrzejfudro
marked this pull request as ready for review
September 16, 2026 10:34
andrzejfudro
enabled auto-merge
September 16, 2026 10:35
janzak-dev
previously approved these changes
Sep 16, 2026
… logging - Call c.wg.Add(1) adjacent to each worker goroutine instead of c.wg.Add(2). - Remove redundant WIP log when PauseSchedulingEnabled is false in cmd/experiment_state_server/server/prod.go. TAG=agy CONV=b35410bc-068d-4e00-bae0-22b3ca180c39
janzak-dev
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This code is currently untested but is behind the feature flag. In parallel I'm working on PR to push events to this stream and I'll verify it then. It is splitted to avoid big PRs as this is mostly boilerplate.
Summary
Implements the background resumption stream consumer loop and resumption worker in the Experiment State Server (ESS) for Task 13 of the Pause Scheduling feature.
Key Changes
xmc-scheduling-resume-streamvia consumer groupess-group, parsing trigger payload (batch_size,max_backlog_threshold), and dispatching toResumptionWorker.Close()to prevent Kubernetes SIGKILLs and manages worker lifecycles with an internalsync.WaitGroupto avoid server shutdown deadlocks.ProdServerbehind thePauseSchedulingEnabledfeature flag and registers teardown in the LIFO cleanup stack while keepingserver/BUILD.bazelfree of Redis client dependencies.Verification
bazel test //cmd/experiment_state_server/resumption/...(Passed)bazel test //cmd/experiment_state_server/server:server_test(Passed)bazel test //cmd/experiment_state_server:health_test(Passed)bazel build //cmd/experiment_state_server:experiment_state_server(Passed)TAG=agy
CONV=b35410bc-068d-4e00-bae0-22b3ca180c39