8000 Fail loudly: stop on missing pools, dropped prompts, and numerical failures instead of skipping by timvieira · Pull Request #4 · rycolab/pool-dpp · GitHub
[go: up one dir, main page]

Skip to content

Fail loudly: stop on missing pools, dropped prompts, and numerical failures instead of skipping - #4

Open
timvieira wants to merge 1 commit into
mainfrom
fail-loudly
Open

timvieira wants to merge 1 commit into
mainfrom
fail-loudly

Conversation

@timvieira
@timvieira timvieira commented Nov 7, 2026 •
Copy link
Copy Markdown
Collaborator

Why

Tim's standing rule: when code hits a failure (a missing input, non-convergence, NaN or inf, a target it cannot reach, an exception), it stops with a clear error. It does not skip the item, retry, fall back, clamp, or return a best effort. A workaround is Tim's decision, made with Manuel, and is disclosed where the results are read.

The code broke this rule in many places, and some of them changed reported numbers:

  • The analysis loops skipped missing pool directories (if not os.path.exists(p): continue). The committed outputs have 30 GSM8K pools and 99 Ambrosia pools, although the scripts and make_figures.sh target 100.
  • math500.py dropped prompts and empty draws.
  • bound_empirical.py dropped every pool and block with Ẑ = 0 from the variance.
  • The redraw in estimation.py also swapped pools after numerical failures, and logged them as "Zhat = 0".

Every number the paper quotes must be checkable, so each such place now stops with an error. The error names the task, setting, pool and prompt, and says what to do. Where the method itself excludes something (the Ẑ = 0 redraw, a block estimate that needs 10 blocks), the trigger is now exact and the count is written to the output JSON. Where the right fix needs no new data (ddof = 1, t quantile, rank-only redraw trigger, counting Ẑ = 0 as a zero, scoring Ancestral sets as drawn), it is implemented, not just raised. Anything else that needs a choice is listed under Decisions below. No opt-in "allow skip" flags were added.

Nothing in the numerical method changes beyond this. Draws use the same random numbers as before: the new checks run after each draw, so a rerun reproduces the committed selections wherever nothing fails.

@manueldeprada, most rows below need a rerun or a decision from you. Tim would like to keep the discussion on this PR.

What changes, and what it means for the committed runs

"Fired?" is judged from the committed analysis/*.json; the caches are not in the repo. "Unknown" means the caches are needed to tell.

Core: src/ and the vLLM patch

file: function before (silent) now fired in committed runs? Manuel
estimation.py: run_kernel_method Redraw triggered on len(selected) < k, so a numerical failure on a full-rank pool (Ẑ > 0) was swapped for another pool and logged as "Zhat = 0" Redraws only when the pool's embeddings span fewer than k dimensions (span_rank, the same test as PoolRedraw.first_valid). A short selection on a full-rank pool raises RuntimeError. A full selection with rank < k raises too (tolerance disagreement) Likely. The paper cites 24 redraws of 9,900 at T'=1. The committed bound rows (bound_raw_ambrosia.json, the same sT=1.0 pools) show only 9 (pool, prompt) pairs with Ẑ = 0. If both counts cover the same pools, up to 15 of the 24 were numerical failures Rerun STAGES=estimate. It checks every stored selection: one not from its own full-rank pool stops with "swapped for another pool". Delete those files and rerun
estimation.py (new write_redraws) Redraws were only printed Writes a sidecar X.redraws.json per PoolDPP-Samp/-MAP/Naive selection file: n_prompts, n_redrawn, each redrawn prompt with its rank, number of unique strings and source pool, and the sampler tier-path counts. It is rebuilt from the stored selections on resume n/a Rerun STAGES=estimate (fast on existing selections) before the figures, which now read the sidecars
estimation.py: iter_prompt_data Prompts whose logps and logps_t1 lengths differ were skipped; for PoolDPP they were then redrawn and logged as Ẑ = 0 Raises ValueError if any of the five arrays differ in length. Prompts with fewer than 2 unique strings are still not yielded, but every caller handles them explicitly Unknown —
estimation.py: PoolRedraw.first_valid A redraw pool without the prompt was skipped, which changes which pool the prompt takes KeyError Unknown —
estimation.py: sample_kmeans_for_prompt, run_kmeans Fewer than K unique strings: returned fewer than K strings, or (1 string) dropped the prompt; an empty cluster was skipped Raises; see Decisions Unknown (likely at low sT on Ambrosia) Decide
estimation.py: greedy_map NaN/+inf log scales gave a short selection, which was then redrawn ValueError. Returns short only when rank < k; RuntimeError when its residual tolerance disagrees with the rank Unknown —
estimation.py: sigma_hat_from_embeddings Prompts with fewer than 2 draws were skipped; an sT with no prompts was left out Raises No (100+ draws per prompt) —
dpp/stable.py: StableKDPP NaN/+inf log scales and non-finite embedding rows were dropped as "no mass". sample() could return a short list on a numerical failure ValueError for those rows (−inf and zero rows are still exact zero mass). [] only when Ẑ = 0; otherwise RuntimeError. inclusion_probs raises instead of returning None on a short SVD Unknown —
dpp/stable.py: tier paths (new .info, kernel_info) (a) rows more than 300 nats below the top were dropped, and (b) the tier-by-tier draw ran, both without any record (a) "drop": records log10_drop_bound, a rigorous upper bound on the left-out probability mass (Fischer and Hadamard inequalities), and does not stop. (b) "tiered": raises unless the gap at every cut exceeds TIER_NATS = 300 nats. Counts go to the sidecars and the points JSON Unknown. The median GSM8K log-weight range is 1,362 nats at T=0.43 (weight_diagnostics.json), so the log-scale range is about 681 nats, above TIER_NATS: both paths can fire Decide (see Decisions 5)
vllm_dual_logprobs.patch A decoding step with no at-temperature log-prob added 0.0 (probability 1). In raw_processed_logprobs mode, TopKTopPSampler still picked FlashInfer when top-k/top-p were set without seeds, and forward_cpu was unpatched; both return no processed log-probs That case makes the total unknown (None), never +0.0. Two new hunks keep the dual mode off FlashInfer and patch forward_cpu. The patch still applies to vllm 0.11.0 with no offset or fuzz. I checked this by applying it to the 0.11.0 sources from GitHub and byte-compiling the result Probably not: the runs set no top-k/top-p, so the native path was used Re-apply the patch before any new sampling
sample_vllm.py: extract_results A missing attribute was stored as NaN (patch not applied), and NaN rows were later dropped by the sampler RuntimeError for missing, None or non-finite log-probs, with the fix; also checks that n completions came back Unknown —
utils/math_utils.py: load_gsm8k_dataset On any error, printed it and returned an empty DataFrame, so callers ran on zero questions Raises No —
det_beam.py: greedy_map_logdet Clamped a zero residual to 1e-300 and kept selecting from a zero-determinant set RuntimeError Unknown —
sample_divbs.py pad = EOS would mask log p(EOS) out of every string Raises for such a tokenizer No (Gemma has a pad token) —
utils/ambrosia_utils.py A golden query that fails to run was skipped (coverage) or its error string was added to the golden rows; a golden list that failed to parse became empty Raises; same for an unknown mode and missing stats inputs Unknown (diagnostic only: --ambrosia-stats) —

Bounds (bound_empirical.py, bound_tables.py)

This branch is rebased on main at f01dba9, which includes PR #2 and the history-note cleanup. PR #3 merges into it cleanly.

function before now fired? Manuel
pool_logzhat, block_relvar, measured_relvar A pool or block with Ẑ = 0 returned None and was left out of Var(Ẑ)/Z², biasing it down Ẑ = 0 enters as a zero (log Ẑ = −inf). New per-prompt sub_nzero[m]; summary block_n_zero_blocks, block_n_prompts, n_prompts_measured_relvar Yes. Ambrosia: 5,910 of 198,000 blocks at m=10 (3.0%), down to 9 whole pools at m = N = 200, which were also left out of measured_relvar. FineWeb: 265 of 20,000 at m=10. GSM8K: none. Every Ambrosia sub_relvar and measured_relvar, and FineWeb m=10, will change Rerun bounds
stable_log_e_K, exact_target A kernel that needs the tiered path counted as Ẑ = 0 in MC draws; the prompt was skipped ("no target, skipped") Raises. Rows record target_tier_path and target_log10_drop_bound No (all 100/50/50 prompts in the committed rows) —
mc_relvar Non-finite MC values were dropped; mean 0 gave NaN Raises No (n_prompts complete; bound_tables regenerates identically) —
extract, pool_files Missing pools, a missing sT and mismatched lengths were skipped; FineWeb globbed whatever pool* dirs existed Raises (all missing pools listed at once); FineWeb uses exactly pool1..9 Committed runs used 99 Ambrosia pools (0..98) and 30 GSM8K pools, so make_figures.sh (--n-pools 100) now stops Provide the pools, or pass the count you mean
summarize.med, bound_tables.q isfinite filters dropped NaN and inf from medians Raises. bound_tables adds n_prompts_measured_across_pool No: on the committed bound_raw_*.json, bound_tables.py gives the same JSON as main's version, and bound_required_n.py the committed one, apart from the new key —
t1_enumeration A case was skipped if its prompt had too few strings Raises No —

Analysis scripts

file before (silent) now fired? Manuel
plot_gsm8k, plot_ambrosia, significance, baselines, baselines_gsm8k, heldout_diversity, length_stratified, length_analysis, pool_ceiling, weight_diagnostics, ess_*, gsm8k_n2500, math500 Missing pool dirs, est dirs, run files and pool files were skipped One FileNotFoundError listing every missing pool, e.g. "gsm8k: 70 of 100 pools missing: pool30..pool99" Yes. GSM8K: 70 of 100 missing (gsm8k_points, gsm8k_instructed_points, baselines_gsm8k*: n_pools = 30). Ambrosia: 1 of 100 (ambrosia_points, baselines_ambrosia, pool_ceiling, recall_vs_ceiling: n_pools = 99) Generate the pools, or decide the counts (Decisions 1)
All selection readers (shared check_selection / check_selections) Skipped non-4-tuple records, prompts not in the gold set, unparsable questions, and sets of the wrong size Each file must cover exactly the task's prompts, once each, with exactly K strings No whole-question drops (cell counts are complete). Short k-Means sets unknown —
Run dirs (all collectors) MAP's runs 1..9 were "skipped" by the same exists check that hid missing runs Explicit deterministic flag: MAP reads run 0; stochastic methods need every run dir, contiguous 0..N-1 Unknown —
nanmean, nanmedian, isfinite filters, logdet → NaN (all scripts) NaN values were dropped from means Plain means plus a check that raises on the item Unlikely (the 1e-6 ridge keeps Grams PD) —
plot_ablations.collect, plot_intrinsic (run mode) np.std with ddof=0 inside a t-interval; missing and NaN runs skipped ddof = 1; missing runs raise Yes for plot_ablations: every stochastic ablation bar is √(10/9) ≈ 1.054× too narrow. plot_intrinsic run mode: not used by make_figures.sh, which uses pool_t Regenerate the ablation figures
beam_baselines.ci95 1.96·sd/√n; nanmean; NaN CI for 1 run t quantile, df = n−1; raises for n < 2; deterministic methods get null CIs Yes: every SBS/CPSBS CI grows 2.262/1.96 = 1.154× at n=10 Rerun all three tasks
beam_baselines (method lists, runs) Missing caches omitted; partial runs truncated Explicit EXPECTED list per task; missing raises Yes: the FineWeb JSON has no DivBS rows Decisions 13
plot_ambrosia, pool_ceiling, beam_baselines (golden queries) A golden query that failed to run was dropped (except: pass), shrinking the recall denominator Raises Unknown (needs ambrosia.csv) Decisions 9
math500._score_pool Cells with fewer than K unique strings were dropped for all methods; empty DPP draws dropped (if sel:); short MAP sets scored Ancestral and the ceiling score every cell, as drawn. k-Means, Naive-DPP and PoolDPP leave out cells with rank < K, and the counts go to math500.json (n_cells_excluded_rank_lt_K, _exclusions) No: the committed math500.json (1dabb9f) has n_cells = 1000 = 10 pools × 100 problems for every method and knob. With no cell excluded, the random stream is unchanged Decisions 8
math500.cmd_verify A math-verify timeout or exception scored the answer as incorrect The error is stored and the run stops, listing them all Unknown (the verify cache is not committed) Decisions 8
embedder_probe n_questions fell back to the item index, so it counts items; builders skipped silently Counts distinct prompts; drop reasons go to a new filters key Yes: the committed n_questions (582 / 468) is the item count (= n_gsm8k / n_sql) The paper must not cite 582 as questions; rerun
pair_judge Retried any error, then recorded no verdict; contradictions took the rule's label Retries HTTP 429/5xx only (4 tries), then raises; contradictions raise Unknown Decisions 11
llm_selection Globbed pools; dropped prompts with fewer than K strings; nanmean Fixed pool list; raises No: the committed llm_selection.json has 500 = 10 pools × 50 prompts, 0 unparsed replies —
length_stratified, length_analysis, weight_diagnostics, gsm8k_n2500, ess_* Uncounted exclusions (fewer than 10 strings, quintile gates, fewer than 2 unique strings) Kept, and counted in the JSON Same numbers; new count keys Rerun for the keys
plot_ndial, dual_vs_primal, embed_instructed Missing settings or keys dropped; nearest grid N used Raise No (ndial_gsm8k.json regenerates byte-identical) —

Decisions for timv/Manuel

Each of these now stops a script, or is left as is and recorded.

  1. Pool counts. GSM8K has 30 pools, but the scripts and make_figures.sh say 100. Sample pools 30–99, or use 30 everywhere and say so in the paper. Ambrosia has 99: which pool is missing, and why? (The bound runs used pool0..98.)
  2. Redraw source pools: spare pools or replicates. 344301b tried 50 spare pools first (N_EXTRA=50). 952d1ac removed N_EXTRA when it raised N_POOLS from 30 to 100, so other_pools in run_ambrosia.sh now redraws from the next replicate pool. A redrawn prompt then shares a pool between two replicates, while the across-pool CIs treat them as independent. The pool layout is not changed here.
  3. Redraw count. The paper's 24 of 9,900 against the 9 Ẑ = 0 pairs in the bound rows (table above). Rerunning dpp-estimation will stop on any stored selection that the old trigger swapped.
  4. k-Means on pools with fewer than K unique strings now raises in dpp-estimation, and every reader rejects short sets. Options: redraw as PoolDPP does, keep the n < K strings and report them, or exclude and count.
  5. StableKDPP tier paths. The sampler's behavior was not changed except for the gap check.
    • (a) drop. The rows within 300 nats of the top span K dimensions, and all lower rows are dropped. This can drop most of the mass near the cut. Example: log scales (0, −299.99×3, −300.01×2), K=3: the exact dropped mass is 0.674, and the recorded bound is 10^0.93. Proposal: stop when log10_drop_bound > log10(2⁻⁵²) ≈ −15.65, i.e. whenever the drop is visible in double precision.
    • (b) tiered. Exact only as the tiers separate. Counterexample: log scales (0, −299.99, −300.01×3), K=3. The tiered draw always contains both top rows, which the k-DPP does with P = 0.497 (old StableKDPP: 1.000). The rule used now: at every cut where the draw continues below the tier, the lowest log scale in the tier must exceed the highest log scale below the cut by more than TIER_NATS = 300. Otherwise RuntimeError; a stored selection whose first cut fails this rule also stops the rerun. The fix is an exact sampler for such kernels. One option: a single factorization over rows within about 700 nats of the top, which a double can hold.
    • (c, d) _merge_duplicates merges rows equal to 9 decimals, and RANK_TOL = 1e-10 decides Ẑ = 0. These are tolerances. They are disclosed here, and the rank of each redrawn prompt goes to the sidecar.
  6. The paper says the experiments use alg:dual-kdpp (the dual sampler); the code uses StableKDPP (graded SVD, merged duplicates, tiers). Change the text or the code.
  7. The exact target in bound_empirical for tiered kernels now raises. It did not fire in the committed runs.
  8. MATH-500.
    • Cells with rank < K are now excluded and counted for k-Means, Naive-DPP and PoolDPP, while the main experiments redraw. So methods are compared over different cell sets. Redraw, or keep this and disclose it.
    • math-verify timeouts and errors now stop the run. Choose a longer deadline or manual checks.
    • The random stream changes only in a run where some cell is excluded. The committed run has none, so a rerun on the same caches reproduces it, provided the verify cache has no errors.
  9. Ambrosia golden queries that fail to run now raise in plot_ambrosia, pool_ceiling, beam_baselines and ambrosia_utils. Fix the data, or exclude the question and count it.
  10. Generated SQL runs on a shared per-question database, so an UPDATE, DELETE or DROP changes later results (plot_ambrosia, pool_ceiling). PRAGMA query_only would make such queries count as misses. Not changed.
  11. Pair judge. "Different answer" or no verdict on a pair the rule calls same-answer now raises. Options: exclude and count, re-ask, or use the rule's label.
  12. llm_selection: fewer than 40 long strings now raises (the old code showed short strings instead).
  13. Beam baselines.
    • No code in this repo produces the SetSMC caches.
    • FineWeb DivBS runs with --chat-template; the other FineWeb runs are raw continuations.
    • baselines.py --task gsm8k and baselines_gsm8k.py both write baselines_gsm8k.json, so the second overwrites the first in make_figures.sh.
  14. Undefined held-out metrics now raise: distinct-n of a set with no n-grams (was NaN, dropped) and Jaccard of two empty strings (was 0.0).
  15. recall_vs_ceiling divides by each question's ceiling averaged over pools, not by the selection's own pool's ceiling. Is that intended?
  16. GSM8K answer extraction (math_utils.extract_model_num) falls back to the last number in the text. That is part of the accuracy metric, so it is not changed but is disclosed here. baselines_gsm8k counts distinct answers from #### lines only, while beam_baselines uses the fallback.

Existing approximations: kept, and disclosed here and in the docstrings

  • A ridge of 1e-6·I in every log-determinant.
  • greedy_map's residual tolerance of 1e-10.
  • TIER_NATS, RANK_TOL, DUP_DECIMALS (Decisions 5).
  • bound_empirical's EIG_FLOOR (1e-8) and the clip of negative rounding eigenvalues.
  • The CP sampler's tail_tol (1e-7) and its 1e-15 tail cutoff (sbs_decoding).
  • DetBS's 1e-12 floor on feature norms.
  • MATH-500 keeps the largest log p of a string seen with several tokenizations; the ESS and N=2500 merges keep the first occurrence.
  • The embedder_probe caps (40 GSM8K strings per prompt, 600 items) and pool_ceiling's 10 subsample draws.
  • The permutation-test p-value floor of 1/n_perm.
  • The ablation "Quality + Diversity" interval adds the two SDs in quadrature (hypot), ignoring their covariance.
  • prompt_sd in beam_baselines uses ddof = 0.
  • MMR maps constant log-probs to 0.

How I tested

  • pytest (testpaths = tests), on the branch rebased on main: 242 passed (136 on main), plus python tests/test_stable.py OK. No GPU, LM, vLLM or network: vLLM, Gemini and dataset loaders are stubbed.
  • New test files: tests/test_fail_loudly_src.py, tests/test_fail_loudly_plots.py, tests/test_fail_loudly_baselines.py, tests/test_fail_loudly_diagnostics.py. They cover one behavior each:
    • every missing-pool and missing-run error, with the full list;
    • a numerical failure on a full-rank pool raising instead of being redrawn;
    • the sidecar counts, identical after a resume;
    • a stored selection swapped by the old trigger being detected;
    • the tier counterexample raising;
    • the drop bound bounding the exact dropped mass, checked by enumeration;
    • Ẑ = 0 blocks entering the variance as zeros;
    • math-verify failures, and MATH-500 counting excluded cells while scoring Ancestral;
    • the patch never adding a default log-prob;
    • ddof = 1 and the t quantile.
  • The patch applies to the vllm 0.11.0 sources with no offset or fuzz, and the patched files byte-compile.
  • On the committed bound_raw_*.json, bound_tables.py gives exactly what main's bound_tables.py gives, plus the new count key. bound_required_n.py reproduces the committed JSON, and plot_ndial.py reproduces ndial_gsm8k.json byte for byte.
  • A trial merge with PR Commit the experiment outputs the paper quotes; CLAUDE.md rule and a checker #3 (commit-paper-outputs) succeeds with no conflicts.
  • Not tested: the scripts on the real caches, which are not in this repo. The "fired?" column is from the committed JSON only.

🤖 Generated with Claude Code

https://claude.ai/code/session_015dhVomjum51FtiFzcbzczN

…ilures instead of skipping

Every place that silently skipped, substituted, clamped or filtered data or a
failure now stops with an error that names the task, setting, pool and prompt
and says what to do. Exclusions that are part of the method stay, with an
exact trigger, and are counted in the output JSON.

Core:
- estimation: the Zhat = 0 redraw triggers on the rank test only; a short
  selection from a full-rank pool raises (it was swapped for another pool and
  logged as Zhat = 0). Each PoolDPP/Naive selection file gets a sidecar
  X.redraws.json with redraw counts and sampler tier paths, rebuilt on resume,
  which also detects stored selections swapped by the old trigger.
- dpp.stable: NaN/+inf log scales raise; [] only for Zhat = 0; the tier paths
  are recorded (drop: rigorous bound on the left-out mass; tiered: raises
  unless the gap at every cut exceeds TIER_NATS).
- vLLM patch: no default 0.0 at-temperature log-prob (unknown -> None); the
  dual mode no longer selects FlashInfer or the unpatched CPU path, which return
  no processed log-probs. sample_vllm and math500 raise on a missing value.
- bound_empirical: pools/blocks with Zhat = 0 enter the variance as zeros
  (they were dropped); prompts needing tiers, non-finite MC values and missing
  pools raise.
- analysis: missing pools, run files and prompts raise; nanmean and isfinite
  filters replaced by checks; ddof=1 in plot_ablations and plot_intrinsic,
  t quantile in beam_baselines; math500 counts its rank < K cells.

Tests: tests/test_fail_loudly_{src,plots,baselines,diagnostics}.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015dhVomjum51FtiFzcbzczN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0