Repository navigation
Conversation
…s and 4x compute does not save it 190/190 cells COMPLETED, 0 invalid. One ladder at the exact frozen S13-S16 contract (194,560 presentations at every width, CVS-4k/D0036, E0004, RAW-768, split RNG streams), plus a control arm with ONE predeclared rule (4x updates) at H=128/256/512. Primary curve (mean test loss, paired steps vs the previous rung): H1 771p 0.015989 | H2 1541p 0.015584 (best, step -0.000406 unresolved) | H4 3081p 0.016280 | H8 6161p 0.017324 (worse) | H16 12321p 0.018735 (worse) | H32 24641p 0.019586 | H64 49281p 0.019585 | H128 98561p 0.020105 | H256 197121p 0.019717 (only non-monotonic step) | H512 394241p 0.020725 The best static loss belongs to the SMALLEST models; marginal gain per added parameter is damaging for every step above H=2 (except an unresolved H=256). The H=16 and H=32 rungs reproduce S14/S15/S16's anchor values exactly, so the curve and the four prior studies are one consistent body of evidence. Control (4x updates) vs the same width's primary, paired by seed: H128 0.020105 -> 0.020951 (+0.000846, strictly WORSE) H256 0.019717 -> 0.019793 (+0.000076 unresolved) H512 0.020725 -> 0.020602 (-0.000123 unresolved) rescued_widths = [] and every control rung is 3.5-4.6 milliloss behind the H=2 knee. So the preregistered dichotomy resolves to neither: capacity beyond ~1.5K parameters is not merely useless but sometimes damaging, and 4x optimizer work does not make it pay. The binding constraint on this contract is the training signal available to the student, not its size - which also explains why every representation and supervision effect in S14-S16 appeared at small widths. Costs scale the wrong way too (H=512: 15.8 s/run, 8.6 MB, worse in every phase slice). Next axis per #17's own trigger: data volume/quality.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
The ladder execution and exact-head CI look sound, but I found three report/provenance issues before merge. No training rerun is needed.
- The result does not identify “training signal/data is the binding constraint.” The preregistered control only tests whether 4× more optimizer work rescues H128/H256/H512. It does not vary corpus size, label quality, regularization, optimizer schedule, or architecture. The observed result is: larger RAW widths are worse under the frozen primary contract, and 4× updates do not rescue the three tested large widths (H128 is strictly worse; H256/H512 unresolved). That supports “capacity scaling is not paying under this contract / within the tested 4× envelope,” not “data is the binding constraint.” Please withdraw the causal data-binding/overfitting claims and frame data-side work as the next hypothesis to test, not a finding.
This also needs to match the preregistration’s own rule: it says a non-rescued control indicates capacity saturation within the 4× envelope. The report currently says the dichotomy resolves to “neither,” which is inconsistent with the frozen analysis contract.
-
The preregistered slice target was the knee + H512, but the analyser hard-codes H32 + H512.
s17_analyse.py::_slices()uses(32, 512)even though the preregistration says “phase/material slices at the knee and at H=512.” Once the observed knee was H2, the required comparison was H2 vs H512. Preserve sealed X0016; add a corrected additive slice artifact/erratum and fix the analyser so the replay computes the declared knee rather than H32. No model rerun is needed. -
Keep H2 as the observed mean optimum / preregistered point-estimate knee, not a resolved threshold. H2 vs H1 is unresolved and H4 vs H2 is unresolved. The strong evidence for degradation begins at the H8 and H16 adjacent steps. So wording such as “capacity beyond ~1.5K is damaging” is stronger than the intervals support; “H2 has the lowest observed mean, with significant worsening at H8/H16 and no rescue of the tested large widths under 4× updates” is defensible.
Small metadata cleanup: the PR body is currently stored as a JSON object string beginning {"body": ...}; please replace it with the Markdown body itself.
Exact-head Actions at 435b665 are independently verified: 234 passed, 3 skipped, 4 warnings.
{"body": "## What this is\n\nS17 / issue #17, the axis S14, S15 and S16 all pointed at, executed to the design you specified: the full RAW ladder under the exact frozen S13\u2013S16 training contract, plus one control to separate capacity from undertraining.\n\nFrozen as X0016 before any outcome: 190/190 cells COMPLETED, 0 INVALID, sealed at
record_hash sha256:bf4307e6\u2026. Preregistrationsha256:ef775882\u2026.\n\n## The design, exactly as authorized\n\n| | primary | control |\n|---|---|---|\n| widths | H = 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 | H = 128, 256, 512 |\n| parameters | 771 \u2192 394,241 (exact, generated bycvslab.families) | 98,561 / 197,121 / 394,241 |\n| student updates | 760 \u00d7 256 = 194,560 presentations at every width | 3,040 = 4\u00d7 \u2014 one constant rule, no per-width tuning, H=128 included as the anchor |\n| fixed | CVS-4k over D0036 (spec hash asserted against X0008's sealed 4k arm), E0004 exam, RAW-768, T0004 optimizer/init, split RNG streams, cold starts | identical |\n\nThe control's predeclared limitation is written into the preregistration: a null separates "capacity saturated" from "compute helped less than 4\u00d7", not from "no compute would help". A proportional-to-parameters rule would have multiplied H=512's run time by ~128.\n\n## The curve turns immediately \u2014 and the turn is not a capacity knee\n\nMean test loss, paired-by-seed steps versus the previous rung (positive = worse):\n\n| H | params | pres/param | loss | step |\n|---:|---:|---:|---:|---:|\n| 1 | 771 | 252.3 | 0.015989 | \u2014 |\n| 2 | 1,541 | 126.3 | 0.015584 | \u22120.000406 (best rung) |\n| 4 | 3,081 | 63.1 | 0.016280 | +0.000697 |\n| 8 | 6,161 | 31.6 | 0.017324 | +0.001044 (worse) |\n| 16 | 12,321 | 15.8 | 0.018735 | +0.001412 (worse) |\n| 32 | 24,641 | 7.9 | 0.019586 | +0.000851 |\n| 64 | 49,281 | 3.9 | 0.019585 | +0.000183 |\n| 128 | 98,561 | 2.0 | 0.020105 | +0.000520 |\n| 256 | 197,121 | 1.0 | 0.019717 | \u22120.000388 (only non-monotonic step) |\n| 512 | 394,241 | 0.5 | 0.020725 | +0.001008 |\n\nThe best static loss belongs to the smallest models. Marginal gain per added parameter is damaging for every step above H=2 except an unresolved H=256; H=512 costs 15.8 s/run and 8.6 MB and is worse than a 1.5K-parameter model in every phase slice (endgame 0.0615 vs 0.0396). The H=16 and H=32 rungs reproduce S14/S15/S16's anchor values exactly, so this curve and the four prior studies are one consistent body of evidence.\n\n## The undertraining test: no width was rescued\n\n| width | primary | 4\u00d7 control | control \u2212 primary | vs the H=2 knee |\n|---:|---:|---:|---|---|\n| 128 | 0.020105 | 0.020951 | +0.000846 (strictly worse) | +0.004631 (worse) |\n| 256 | 0.019717 | 0.019793 | +0.000076 (unresolved) | +0.003472 (worse) |\n| 512 | 0.020725 | 0.020602 | \u22120.000123 (unresolved) | +0.004281 (worse) |\n\nrescued_widths: []. Quadrupling the optimizer work left every large rung far behind the 1,541-parameter model, and at H=128 made it strictly worse \u2014 which is what overfitting looks like when the data is the binding constraint.\n\n## What this means (the answer to your question)\n\nNeither branch of the preregistered dichotomy: capacity beyond ~1.5K parameters does not help this evaluator and 4\u00d7 the compute does not make it help. The binding constraint is the training signal available to the student (18,953 rows of CVS-4k supervision on this population), not the student's size. That explains rather than contradicts the earlier studies \u2014 S14's "RAW degrades from 3K to 25K" was the leading edge of this whole-ladder phenomenon, and every representation/supervision effect in S14\u2013S16 appeared at small widths, where this curve says the headroom is.\n\n#17's own follow-up trigger fires: the curve is flat well before H=512, so scaling raw width is not where the static program should spend compute. Next static axis: data volume/quality \u2014 including the selective-labeling/coverage question (#16). No engine-search promotion is implied.\n\n## Non-claims\n\nNo search, game or strength claims; static losses on one exam. One population, one teacher, one optimizer, one fixed-update regime, one machine. The control envelope is 4\u00d7, not unlimited compute. H=256's agreement with the historical ~200K parameter scale is a comparison point, not inherited truth. Larger rungs have 10 seeds (smaller 20, the S13\u2013S16 list), so intervals are wider above H=32.\n\n## Artifacts\n\ndocs/S17_SCALING_LADDER.md;tools/s17/(preregistration + hash, generated ladder and control rule, driver, analysis, state, result, summary); sealedX0016(190/190). Localpytest: 237 passed.\n"}