8000 feat(rod-backend): resolve generic resources to specific accelerators by janzak-dev · Pull Request #605 · google/xmc · GitHub
[go: up one dir, main page]

Skip to content

feat(rod-backend): resolve generic resources to specific accelerators - #605

Merged
janzak-dev merged 22 commits into
mainfrom
feat/rod-specific-accelerators
Aug 27, 2026
Merged

janzak-dev merged 22 commits into
mainfrom
feat/rod-specific-accelerators

Conversation

@janzak-dev
@janzak-dev janzak-dev commented Aug 26, 2026 •
Copy link
Copy Markdown
Collaborator

Summary

Resolves generic accelerator resources (nvidia.com/gpu, google.com/tpu) in Resource Observability Dashboard (ROD) queue overviews to compound specific accelerator models (e.g., nvidia.com/gpu: nvidia-tesla-a100, google.com/tpu: v5p-8) using multi-label GKE node metadata and ResourceFlavor naming fallbacks.

Motivation & Root Cause

Previously, Kueue resource resolution relied solely on a single hardcoded node label key (cloud.google.com/gke-accelerator) when attempting to resolve generic accelerator resources. ClusterQueues using other standard node labels (e.g., cloud.google.com/gke-gpu-type, cloud.google.com/gke-tpu-accelerator, kueue.x-k8s.io/gpack-gpu, gpu.nvidia.com/class) or relying on the ResourceFlavor name itself were not being resolved, resulting in generic nvidia.com/gpu or google.com/tpu classification in queue capacity overviews.

Key Changes

  1. Multi-Label Accelerator Resolution (cmd/rod_data_provider/kueue_resource_extractor.go):

    • Iterates through standard GKE and Kueue accelerator candidate node labels (cloud.google.com/gke-accelerator, cloud.google.com/gke-gpu-type, cloud.google.com/gke-tpu-accelerator, kueue.x-k8s.io/gpack-gpu, gpu.nvidia.com/class).
    • Excludes topology geometry strings (cloud.google.com/gke-tpu-topology, e.g. 2x2x1, 2x4) from hardware model resolution.
    • Formats resolved accelerators with compound delimited domain prefixes (nvidia.com/gpu: <model> or google.com/tpu: <model>).
    • Falls back to ResourceFlavor.Name if isSpecificAcceleratorName(flavorName) matches concrete hardware while excluding generic flavor names (spot, ondemand, default, standard, fallback, generic, shared, custom).
  2. Shared Accelerator Classifier (internal/common/resourceaccumulator/classifier.go):

    • Implements zero-allocation byte-scanner MatchesShortAcceleratorKeyword to prevent false positive substring matches on short 2-character keywords (e.g., pool4, level4, server6).
    • Provides comprehensive classification helpers (IsSpecificAcceleratorName, IsSpecificGPUKey, IsSpecificTPUKey, IsAcceleratorResource).
  3. Unit Test Matrix & Test Deduplication:

    • Added table-driven unit test coverage in kueue_resource_extractor_test.go and classifier_test.go.

Screenshots

We can have both generic and specific accelerators in the list

image

Selecting generic accelerator brings both specific and generic ones

image

Selecting specific accelerators brings only specific ones

image

Stacked Frontend Changes

Frontend hierarchical filtering and picker sorting are stacked in PR #643:
#643

TAG=agy
CONV=acf882de-1561-404c-ba2e-ae0940451165

@gemini-code-assist gemini-code-assist Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements accelerator resource resolution and normalization, allowing generic GPU and TPU resource counts to be merged into specific accelerator types when present. The feedback highlights a critical concurrency issue where calling Normalize() inside getter methods mutates internal state, potentially causing data races under concurrent access. To resolve this, the reviewer suggests making the getters read-only and normalizing eagerly during mutating operations. Additionally, several optimization opportunities were identified to reduce memory allocations by moving local slices to package-level variables and avoiding slice allocations in normalization helpers.

…or normalization and concurrent read safety

- Remove lazy Normalize() calls from read-only getter methods (Totals(), IntTotals(), Order()) to ensure thread safety during concurrent reads, and invoke Normalize() eagerly in Add().
- Move gpuKeywords, tpuKeywords, and acceleratorKeywords slice definitions to package level to avoid allocations.
- Direct-check generic GPU and TPU keys in normalizeGenericGPU and normalizeGenericTPU without slice allocations.
- Add unit test for concurrent getter calls and additional generic normalization test cases.

TAG=agy
…ccelerators control flow, and nil receiver safety
…etes common constants and add classifier unit tests
@janzak-dev
janzak-dev marked this pull request as ready for review August 27, 2026 08:41
@janzak-dev janzak-dev added the priority::2-high top of mind. should not wait. label Aug 27, 2026
@janzak-dev
janzak-dev requested review from andrzejfudro and removed request for AntonTyb and andrzejfudro August 27, 2026 09:03
@janzak-dev
janzak-dev requested a review from AntonTyb August 27, 2026 09:03
@janzak-dev
janzak-dev added this pull request to the merge queue Aug 27, 2026
Merged via the queue into main with commit b8ac319 Aug 27, 2026
13 checks passed
@janzak-dev
janzak-dev deleted the feat/rod-specific-accelerators branch August 27, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority::2-high top of mind. should not wait.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0