Repository navigation
fix(kueue): bound a submitted job's request to what a node can supply - #916
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request introduces LimitRange resources for both the cpu-queue and gpu-queue namespaces to define default container resource requests and maximum limits. The review feedback recommends making these hardcoded resource requests and limits configurable via Helm's values.yaml to allow easier customization across different environments and node shapes.
4a275a9 to
9c41ae7
Compare
A container that names no CPU or memory request is admitted against quota as if it were free, so a queue can admit more work than its nodes can hold. The `LimitRange` supplies a request, and a `max` rejects a container larger than any single node the queue can reach. The values live in `queues` in `values.yaml`, because a deployment's node shapes are its own. Sent with ❤️ from Eka Robotics.
9c41ae7 to
05dab64
Compare
Adds a
LimitRangeto each queue namespace, so every submitted container gets a default CPU and memory request and a ceiling no single node could host. Both are settable underqueuesinvalues.yaml.Without a request, a job is admitted as BestEffort and the kubelet evicts it first under node pressure. We lost a trainer 25 minutes in, on a node reporting 10.8 GiB in use against a request of zero. The client cannot send a request itself, because
ResourceSetputscpu_resourcesandgpu_resourcesin aoneof, so a job asking for an accelerator cannot also ask for CPU or memory. TheLimitRangesupplies what the client could not. It reaches only containers naming no request of their own, which is every job XMC submits, and sets no limit, so a job may still burst above the request rather than being OOM-killed at it.Without a ceiling, the opposite failure. Kueue admits against quota, and quota is a sum over the fleet, so a container asking for more than any single node has can pass admission, hold that quota, and sit
Pendingforever because nothing can host it.maxrejects it at submission instead.That ceiling is coarse by construction. A namespace spanning pools of different sizes can only be given the larger, and allocatable is below capacity in any case, so it rules out what nothing could ever host rather than promising a fit.
The numbers are ours and a deployment with different node shapes will want its own, which is why they are values rather than constants. The in-template defaults are unchanged, so a deployment that sets nothing renders exactly as before.
Tested with
helm lint, the chart configuration tests, andkubectl apply --dry-run=serveragainst a live cluster for both rendered objects.Sent with ❤️ from Eka Robotics.