8000 Bounded per-host concurrency for data collection · Issue #17 · dominikzorgnotti/esx-hcl-check · GitHub
[go: up one dir, main page]

Skip to content

Bounded per-host concurrency for data collection #17

Closed

Description

@dominikzorgnotti

Summary

Host data collection in collectVSphereData is sequential: each host is an independent vCenter round-trip (property retrieve + optional 9.1 SOAP call), so a large environment (100+ hosts) waits on them one at a time. With #4 (cross-host query dedup) already cutting the phase-2 network cost, phase-1 collection is now the dominant latency. This bounds-parallelizes it.

Design

  • Add a -workers flag (default 8) to cap concurrency; -workers=1 forces fully sequential for constrained/rate-sensitive environments
  • Enumerate host jobs sequentially (the govmomi finder is stateful), then run extraction through a bounded worker pool writing into pre-indexed slots
  • Deterministic output preserved: results are placed by enumeration index, so ordering is identical to the sequential version — the only observable change is speed (keeps this non-breaking)
  • Extract a small runBounded(n, workers, fn) primitive so the concurrency is unit- and race-testable without a live vCenter
  • Add -race to the CI go test step so the synchronization is actually verified (the roadmap calls for landing concurrency under the race detector)

Safety notes

  • Per-host extraction writes only to its own slot and its local struct; shared state (client, property collector, exclude config) is read-only. Concurrent one-shot RetrieveProperties/SOAP over the shared soap client is safe.
  • Bounded pool + the v0.2.0 retry/backoff keep us from overwhelming vCenter or the Broadcom API.

Acceptance criteria

  • Output (text + JSON) is identical to sequential for the same environment, including host ordering
  • -workers controls the pool; -workers=1 is sequential
  • go test -race passes and runs in CI
  • Non-breaking; no schema change

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0