Repository navigation
Conversation
…ilures instead of skipping
Every place that silently skipped, substituted, clamped or filtered data or a
failure now stops with an error that names the task, setting, pool and prompt
and says what to do. Exclusions that are part of the method stay, with an
exact trigger, and are counted in the output JSON.
Core:
- estimation: the Zhat = 0 redraw triggers on the rank test only; a short
selection from a full-rank pool raises (it was swapped for another pool and
logged as Zhat = 0). Each PoolDPP/Naive selection file gets a sidecar
X.redraws.json with redraw counts and sampler tier paths, rebuilt on resume,
which also detects stored selections swapped by the old trigger.
- dpp.stable: NaN/+inf log scales raise; [] only for Zhat = 0; the tier paths
are recorded (drop: rigorous bound on the left-out mass; tiered: raises
unless the gap at every cut exceeds TIER_NATS).
- vLLM patch: no default 0.0 at-temperature log-prob (unknown -> None); the
dual mode no longer selects FlashInfer or the unpatched CPU path, which return
no processed log-probs. sample_vllm and math500 raise on a missing value.
- bound_empirical: pools/blocks with Zhat = 0 enter the variance as zeros
(they were dropped); prompts needing tiers, non-finite MC values and missing
pools raise.
- analysis: missing pools, run files and prompts raise; nanmean and isfinite
filters replaced by checks; ddof=1 in plot_ablations and plot_intrinsic,
t quantile in beam_baselines; math500 counts its rank < K cells.
Tests: tests/test_fail_loudly_{src,plots,baselines,diagnostics}.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015dhVomjum51FtiFzcbzczN
timvieira
force-pushed
the
fail-loudly
branch
from
November 7, 2026 03:54
ee50d51 to
40ead1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Tim's standing rule: when code hits a failure (a missing input, non-convergence, NaN or inf, a target it cannot reach, an exception), it stops with a clear error. It does not skip the item, retry, fall back, clamp, or return a best effort. A workaround is Tim's decision, made with Manuel, and is disclosed where the results are read.
The code broke this rule in many places, and some of them changed reported numbers:
if not os.path.exists(p): continue). The committed outputs have 30 GSM8K pools and 99 Ambrosia pools, although the scripts andmake_figures.shtarget 100.math500.pydropped prompts and empty draws.bound_empirical.pydropped every pool and block with Ẑ = 0 from the variance.estimation.pyalso swapped pools after numerical failures, and logged them as "Zhat = 0".Every number the paper quotes must be checkable, so each such place now stops with an error. The error names the task, setting, pool and prompt, and says what to do. Where the method itself excludes something (the Ẑ = 0 redraw, a block estimate that needs 10 blocks), the trigger is now exact and the count is written to the output JSON. Where the right fix needs no new data (ddof = 1, t quantile, rank-only redraw trigger, counting Ẑ = 0 as a zero, scoring Ancestral sets as drawn), it is implemented, not just raised. Anything else that needs a choice is listed under Decisions below. No opt-in "allow skip" flags were added.
Nothing in the numerical method changes beyond this. Draws use the same random numbers as before: the new checks run after each draw, so a rerun reproduces the committed selections wherever nothing fails.
@manueldeprada, most rows below need a rerun or a decision from you. Tim would like to keep the discussion on this PR.
What changes, and what it means for the committed runs
"Fired?" is judged from the committed
analysis/*.json; the caches are not in the repo. "Unknown" means the caches are needed to tell.Core:
src/and the vLLM patchestimation.py: run_kernel_methodlen(selected) < k, so a numerical failure on a full-rank pool (Ẑ > 0) was swapped for another pool and logged as "Zhat = 0"span_rank, the same test asPoolRedraw.first_valid). A short selection on a full-rank pool raises RuntimeError. A full selection with rank < k raises too (tolerance disagreement)bound_raw_ambrosia.json, the same sT=1.0 pools) show only 9 (pool, prompt) pairs with Ẑ = 0. If both counts cover the same pools, up to 15 of the 24 were numerical failuresSTAGES=estimate. It checks every stored selection: one not from its own full-rank pool stops with "swapped for another pool". Delete those files and rerunestimation.py(newwrite_redraws)X.redraws.jsonper PoolDPP-Samp/-MAP/Naive selection file:n_prompts,n_redrawn, each redrawn prompt with its rank, number of unique strings and source pool, and the sampler tier-path counts. It is rebuilt from the stored selections on resumeSTAGES=estimate(fast on existing selections) before the figures, which now read the sidecarsestimation.py: iter_prompt_datalogpsandlogps_t1lengths differ were skipped; for PoolDPP they were then redrawn and logged as Ẑ = 0estimation.py: PoolRedraw.first_validestimation.py: sample_kmeans_for_prompt,run_kmeansestimation.py: greedy_mapestimation.py: sigma_hat_from_embeddingsdpp/stable.py: StableKDPPsample()could return a short list on a numerical failure[]only when Ẑ = 0; otherwise RuntimeError.inclusion_probsraises instead of returning None on a short SVDdpp/stable.py: tier paths (new.info,kernel_info)log10_drop_bound, a rigorous upper bound on the left-out probability mass (Fischer and Hadamard inequalities), and does not stop. (b) "tiered": raises unless the gap at every cut exceeds TIER_NATS = 300 nats. Counts go to the sidecars and the points JSONweight_diagnostics.json), so the log-scale range is about 681 nats, above TIER_NATS: both paths can firevllm_dual_logprobs.patchraw_processed_logprobsmode,TopKTopPSamplerstill picked FlashInfer when top-k/top-p were set without seeds, andforward_cpuwas unpatched; both return no processed log-probsforward_cpu. The patch still applies to vllm 0.11.0 with no offset or fuzz. I checked this by applying it to the 0.11.0 sources from GitHub and byte-compiling the resultsample_vllm.py: extract_resultsutils/math_utils.py: load_gsm8k_datasetdet_beam.py: greedy_map_logdetsample_divbs.pyutils/ambrosia_utils.py--ambrosia-stats)Bounds (
bound_empirical.py,bound_tables.py)This branch is rebased on
mainat f01dba9, which includes PR #2 and the history-note cleanup. PR #3 merges into it cleanly.pool_logzhat,block_relvar,measured_relvarsub_nzero[m]; summaryblock_n_zero_blocks,block_n_prompts,n_prompts_measured_relvarmeasured_relvar. FineWeb: 265 of 20,000 at m=10. GSM8K: none. Every Ambrosiasub_relvarandmeasured_relvar, and FineWeb m=10, will changeboundsstable_log_e_K,exact_targettarget_tier_pathandtarget_log10_drop_boundmc_relvarbound_tablesregenerates identically)extract,pool_filespool*dirs existedmake_figures.sh(--n-pools 100) now stopssummarize.med,bound_tables.qbound_tablesaddsn_prompts_measured_across_poolbound_raw_*.json,bound_tables.pygives the same JSON asmain's version, andbound_required_n.pythe committed one, apart from the new keyt1_enumerationAnalysis scripts
plot_gsm8k,plot_ambrosia,significance,baselines,baselines_gsm8k,heldout_diversity,length_stratified,length_analysis,pool_ceiling,weight_diagnostics,ess_*,gsm8k_n2500,math500gsm8k_points,gsm8k_instructed_points,baselines_gsm8k*: n_pools = 30). Ambrosia: 1 of 100 (ambrosia_points,baselines_ambrosia,pool_ceiling,recall_vs_ceiling: n_pools = 99)check_selection/check_selections)existscheck that hid missing runsdeterministicflag: MAP reads run 0; stochastic methods need every run dir, contiguous 0..N-1nanmean,nanmedian, isfinite filters,logdet → NaN(all scripts)plot_ablations.collect,plot_intrinsic(run mode)np.stdwith ddof=0 inside a t-interval; missing and NaN runs skippedplot_ablations: every stochastic ablation bar is √(10/9) ≈ 1.054× too narrow.plot_intrinsicrun mode: not used bymake_figures.sh, which usespool_tbeam_baselines.ci95nullCIsbeam_baselines(method lists, runs)EXPECTEDlist per task; missing raisesplot_ambrosia,pool_ceiling,beam_baselines(golden queries)except: pass), shrinking the recall denominatorambrosia.csv)math500._score_poolif sel:); short MAP sets scoredmath500.json(n_cells_excluded_rank_lt_K,_exclusions)math500.json(1dabb9f) has n_cells = 1000 = 10 pools × 100 problems for every method and knob. With no cell excluded, the random stream is unchangedmath500.cmd_verifyembedder_proben_questionsfell back to the item index, so it counts items; builders skipped silentlyfilterskeyn_questions(582 / 468) is the item count (=n_gsm8k/n_sql)pair_judgellm_selectionllm_selection.jsonhas 500 = 10 pools × 50 prompts, 0 unparsed replieslength_stratified,length_analysis,weight_diagnostics,gsm8k_n2500,ess_*plot_ndial,dual_vs_primal,embed_instructedndial_gsm8k.jsonregenerates byte-identical)Decisions for timv/Manuel
Each of these now stops a script, or is left as is and recorded.
make_figures.shsay 100. Sample pools 30–99, or use 30 everywhere and say so in the paper. Ambrosia has 99: which pool is missing, and why? (The bound runs used pool0..98.)N_EXTRA=50). 952d1ac removedN_EXTRAwhen it raisedN_POOLSfrom 30 to 100, soother_poolsinrun_ambrosia.shnow redraws from the next replicate pool. A redrawn prompt then shares a pool between two replicates, while the across-pool CIs treat them as independent. The pool layout is not changed here.dpp-estimationwill stop on any stored selection that the old trigger swapped.dpp-estimation, and every reader rejects short sets. Options: redraw as PoolDPP does, keep the n < K strings and report them, or exclude and count.log10_drop_bound> log10(2⁻⁵²) ≈ −15.65, i.e. whenever the drop is visible in double precision._merge_duplicatesmerges rows equal to 9 decimals, andRANK_TOL= 1e-10 decides Ẑ = 0. These are tolerances. They are disclosed here, and the rank of each redrawn prompt goes to the sidecar.StableKDPP(graded SVD, merged duplicates, tiers). Change the text or the code.bound_empiricalfor tiered kernels now raises. It did not fire in the committed runs.plot_ambrosia,pool_ceiling,beam_baselinesandambrosia_utils. Fix the data, or exclude the question and count it.plot_ambrosia,pool_ceiling).PRAGMA query_onlywould make such queries count as misses. Not changed.llm_selection: fewer than 40 long strings now raises (the old code showed short strings instead).--chat-template; the other FineWeb runs are raw continuations.baselines.py --task gsm8kandbaselines_gsm8k.pyboth writebaselines_gsm8k.json, so the second overwrites the first inmake_figures.sh.recall_vs_ceilingdivides by each question's ceiling averaged over pools, not by the selection's own pool's ceiling. Is that intended?math_utils.extract_model_num) falls back to the last number in the text. That is part of the accuracy metric, so it is not changed but is disclosed here.baselines_gsm8kcounts distinct answers from####lines only, whilebeam_baselinesuses the fallback.Existing approximations: kept, and disclosed here and in the docstrings
greedy_map's residual tolerance of 1e-10.TIER_NATS,RANK_TOL,DUP_DECIMALS(Decisions 5).bound_empirical'sEIG_FLOOR(1e-8) and the clip of negative rounding eigenvalues.tail_tol(1e-7) and its 1e-15 tail cutoff (sbs_decoding).embedder_probecaps (40 GSM8K strings per prompt, 600 items) andpool_ceiling's 10 subsample draws.hypot), ignoring their covariance.prompt_sdinbeam_baselinesuses ddof = 0.How I tested
pytest(testpaths = tests), on the branch rebased onmain: 242 passed (136 onmain), pluspython tests/test_stable.pyOK. No GPU, LM, vLLM or network: vLLM, Gemini and dataset loaders are stubbed.tests/test_fail_loudly_src.py,tests/test_fail_loudly_plots.py,tests/test_fail_loudly_baselines.py,tests/test_fail_loudly_diagnostics.py. They cover one behavior each:bound_raw_*.json,bound_tables.pygives exactly whatmain'sbound_tables.pygives, plus the new count key.bound_required_n.pyreproduces the committed JSON, andplot_ndial.pyreproducesndial_gsm8k.jsonbyte for byte.commit-paper-outputs) succeeds with no conflicts.🤖 Generated with Claude Code
https://claude.ai/code/session_015dhVomjum51FtiFzcbzczN