Repository navigation
Prelude QC steps pay full Batch cold-start for ~1s of work #164
Description
The three PRELUDE_SUMMARY_* processes each run as separate awsbatch jobs in the pandas container, and on a real run each one takes over five minutes end-to-end despite only ~1 second of actual compute — the rest is EC2 provisioning and image pull. Since these are trivial process_single QC steps (and SUMMARY_XML fires once per cycle), the cold-start overhead dominates and adds up fast on multi-cycle samples. It would be worth either collapsing them into a single process or running them on the head node so we're not spinning up a fresh oversized instance for each one.
Activity