8000 S17: RAW scaling ladder — best loss at 1,541 parameters, and 4× compute rescues nothing (X0016 sealed, 190/190) by Mnehmos · Pull Request #48 · Mnehmos/chess-vision-studio-lab · GitHub
[go: up one dir, main page]

Skip to content

S17: RAW scaling ladder — best loss at 1,541 parameters, and 4× compute rescues nothing (X0016 sealed, 190/190) - #48

Open
Mnehmos wants to merge 1 commit into
mainfrom
feat/s17-scaling-ladder
Open

Mnehmos wants to merge 1 commit into
mainfrom
feat/s17-scaling-ladder

Conversation

@Mnehmos
@Mnehmos Mnehmos commented Sep 18, 2026
Copy link
Copy Markdown
Owner

{"body": "## What this is\n\nS17 / issue #17, the axis S14, S15 and S16 all pointed at, executed to the design you specified: the full RAW ladder under the exact frozen S13\u2013S16 training contract, plus one control to separate capacity from undertraining.\n\nFrozen as X0016 before any outcome: 190/190 cells COMPLETED, 0 INVALID, sealed at record_hash sha256:bf4307e6\u2026. Preregistration sha256:ef775882\u2026.\n\n## The design, exactly as authorized\n\n| | primary | control |\n|---|---|---|\n| widths | H = 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 | H = 128, 256, 512 |\n| parameters | 771 \u2192 394,241 (exact, generated by cvslab.families) | 98,561 / 197,121 / 394,241 |\n| student updates | 760 \u00d7 256 = 194,560 presentations at every width | 3,040 = 4\u00d7 \u2014 one constant rule, no per-width tuning, H=128 included as the anchor |\n| fixed | CVS-4k over D0036 (spec hash asserted against X0008's sealed 4k arm), E0004 exam, RAW-768, T0004 optimizer/init, split RNG streams, cold starts | identical |\n\nThe control's predeclared limitation is written into the preregistration: a null separates "capacity saturated" from "compute helped less than 4\u00d7", not from "no compute would help". A proportional-to-parameters rule would have multiplied H=512's run time by ~128.\n\n## The curve turns immediately \u2014 and the turn is not a capacity knee\n\nMean test loss, paired-by-seed steps versus the previous rung (positive = worse):\n\n| H | params | pres/param | loss | step |\n|---:|---:|---:|---:|---:|\n| 1 | 771 | 252.3 | 0.015989 | \u2014 |\n| 2 | 1,541 | 126.3 | 0.015584 | \u22120.000406 (best rung) |\n| 4 | 3,081 | 63.1 | 0.016280 | +0.000697 |\n| 8 | 6,161 | 31.6 | 0.017324 | +0.001044 (worse) |\n| 16 | 12,321 | 15.8 | 0.018735 | +0.001412 (worse) |\n| 32 | 24,641 | 7.9 | 0.019586 | +0.000851 |\n| 64 | 49,281 | 3.9 | 0.019585 | +0.000183 |\n| 128 | 98,561 | 2.0 | 0.020105 | +0.000520 |\n| 256 | 197,121 | 1.0 | 0.019717 | \u22120.000388 (only non-monotonic step) |\n| 512 | 394,241 | 0.5 | 0.020725 | +0.001008 |\n\nThe best static loss belongs to the smallest models. Marginal gain per added parameter is damaging for every step above H=2 except an unresolved H=256; H=512 costs 15.8 s/run and 8.6 MB and is worse than a 1.5K-parameter model in every phase slice (endgame 0.0615 vs 0.0396). The H=16 and H=32 rungs reproduce S14/S15/S16's anchor values exactly, so this curve and the four prior studies are one consistent body of evidence.\n\n## The undertraining test: no width was rescued\n\n| width | primary | 4\u00d7 control | control \u2212 primary | vs the H=2 knee |\n|---:|---:|---:|---|---|\n| 128 | 0.020105 | 0.020951 | +0.000846 (strictly worse) | +0.004631 (worse) |\n| 256 | 0.019717 | 0.019793 | +0.000076 (unresolved) | +0.003472 (worse) |\n| 512 | 0.020725 | 0.020602 | \u22120.000123 (unresolved) | +0.004281 (worse) |\n\nrescued_widths: []. Quadrupling the optimizer work left every large rung far behind the 1,541-parameter model, and at H=128 made it strictly worse \u2014 which is what overfitting looks like when the data is the binding constraint.\n\n## What this means (the answer to your question)\n\nNeither branch of the preregistered dichotomy: capacity beyond ~1.5K parameters does not help this evaluator and 4\u00d7 the compute does not make it help. The binding constraint is the training signal available to the student (18,953 rows of CVS-4k supervision on this population), not the student's size. That explains rather than contradicts the earlier studies \u2014 S14's "RAW degrades from 3K to 25K" was the leading edge of this whole-ladder phenomenon, and every representation/supervision effect in S14\u2013S16 appeared at small widths, where this curve says the headroom is.\n\n#17's own follow-up trigger fires: the curve is flat well before H=512, so scaling raw width is not where the static program should spend compute. Next static axis: data volume/quality \u2014 including the selective-labeling/coverage question (#16). No engine-search promotion is implied.\n\n## Non-claims\n\nNo search, game or strength claims; static losses on one exam. One population, one teacher, one optimizer, one fixed-update regime, one machine. The control envelope is 4\u00d7, not unlimited compute. H=256's agreement with the historical ~200K parameter scale is a comparison point, not inherited truth. Larger rungs have 10 seeds (smaller 20, the S13\u2013S16 list), so intervals are wider above H=32.\n\n## Artifacts\n\ndocs/S17_SCALING_LADDER.md; tools/s17/ (preregistration + hash, generated ladder and control rule, driver, analysis, state, result, summary); sealed X0016 (190/190). Local pytest: 237 passed.\n"}

…s and 4x compute does not save it

190/190 cells COMPLETED, 0 invalid. One ladder at the exact frozen S13-S16 contract
(194,560 presentations at every width, CVS-4k/D0036, E0004, RAW-768, split RNG streams),
plus a control arm with ONE predeclared rule (4x updates) at H=128/256/512.

Primary curve (mean test loss, paired steps vs the previous rung):
  H1   771p 0.015989 | H2  1541p 0.015584 (best, step -0.000406 unresolved) |
  H4  3081p 0.016280 | H8  6161p 0.017324 (worse) | H16 12321p 0.018735 (worse) |
  H32 24641p 0.019586 | H64 49281p 0.019585 | H128 98561p 0.020105 |
  H256 197121p 0.019717 (only non-monotonic step) | H512 394241p 0.020725
The best static loss belongs to the SMALLEST models; marginal gain per added parameter is
damaging for every step above H=2 (except an unresolved H=256). The H=16 and H=32 rungs
reproduce S14/S15/S16's anchor values exactly, so the curve and the four prior studies are
one consistent body of evidence.

Control (4x updates) vs the same width's primary, paired by seed:
  H128 0.020105 -> 0.020951 (+0.000846, strictly WORSE)
  H256 0.019717 -> 0.019793 (+0.000076 unresolved)
  H512 0.020725 -> 0.020602 (-0.000123 unresolved)
rescued_widths = [] and every control rung is 3.5-4.6 milliloss behind the H=2 knee.

So the preregistered dichotomy resolves to neither: capacity beyond ~1.5K parameters is
not merely useless but sometimes damaging, and 4x optimizer work does not make it pay. The
binding constraint on this contract is the training signal available to the student, not
its size - which also explains why every representation and supervision effect in S14-S16
appeared at small widths. Costs scale the wrong way too (H=512: 15.8 s/run, 8.6 MB, worse
in every phase slice). Next axis per #17's own trigger: data volume/quality.
@coderabbitai
coderabbitai Bot commented Sep 18, 2026
Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 501938a5-35f1-4661-9430-4197632647fa


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector
Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@Mnehmos Mnehmos left a comment
Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ladder execution and exact-head CI look sound, but I found three report/provenance issues before merge. No training rerun is needed.

  1. The result does not identify “training signal/data is the binding constraint.” The preregistered control only tests whether 4× more optimizer work rescues H128/H256/H512. It does not vary corpus size, label quality, regularization, optimizer schedule, or architecture. The observed result is: larger RAW widths are worse under the frozen primary contract, and 4× updates do not rescue the three tested large widths (H128 is strictly worse; H256/H512 unresolved). That supports “capacity scaling is not paying under this contract / within the tested 4× envelope,” not “data is the binding constraint.” Please withdraw the causal data-binding/overfitting claims and frame data-side work as the next hypothesis to test, not a finding.

This also needs to match the preregistration’s own rule: it says a non-rescued control indicates capacity saturation within the 4× envelope. The report currently says the dichotomy resolves to “neither,” which is inconsistent with the frozen analysis contract.

  1. The preregistered slice target was the knee + H512, but the analyser hard-codes H32 + H512. s17_analyse.py::_slices() uses (32, 512) even though the preregistration says “phase/material slices at the knee and at H=512.” Once the observed knee was H2, the required comparison was H2 vs H512. Preserve sealed X0016; add a corrected additive slice artifact/erratum and fix the analyser so the replay computes the declared knee rather than H32. No model rerun is needed.

  2. Keep H2 as the observed mean optimum / preregistered point-estimate knee, not a resolved threshold. H2 vs H1 is unresolved and H4 vs H2 is unresolved. The strong evidence for degradation begins at the H8 and H16 adjacent steps. So wording such as “capacity beyond ~1.5K is damaging” is stronger than the intervals support; “H2 has the lowest observed mean, with significant worsening at H8/H16 and no rescue of the tested large widths under 4× updates” is defensible.

Small metadata cleanup: the PR body is currently stored as a JSON object string beginning {"body": ...}; please replace it with the Markdown body itself.

Exact-head Actions at 435b665 are independently verified: 234 passed, 3 skipped, 4 warnings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0