Repository navigation
read: score held-out at the reading window, not the ceiling - #6
Merged
Merged
Conversation
cmd_read built its FolderEvaluator with cfg.block and never moved it, so held-out was read at the ceiling while the model read at its window - the opposite of what the comment above it says it does. `stream` already sets evaluator.context as its window grows. Measured on the large Strix profile at step 16,335 (window 2,367 of 8,192), the same held-out text: at the window 1.4164 +/-0.0151 at the ceiling 2.2843 +/-0.0280 every subject 0.70-1.12 nats worse at the ceiling. That flattened the reported held-out for a whole night while training loss fell 1.57 -> 1.29, and since growth's HONEST brake compares train with held-out (max_gap 0.4) it likely kept growth shut: the pool sat at 135 experts. The evaluator now starts at the window the run resumes at and is set to the current window before every evaluation - before, each sample round, after. Runs whose window has reached its ceiling, the released model's among them, score exactly as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
train.py readscored held-out at the context ceiling rather than at the window the model is reading at, which is the opposite of what the comment above the evaluator says. It built itsFolderEvaluatorwithcfg.blockand never updated it. Thestreamcommand already setsevaluator.contextas its window grows (train.py:254, 352);readdidn't.Measured on the large Strix profile at step 16,335 (window 2,367 of 8,192), scoring the same held-out text, 60 chunks per subject:
What it affected:
max_gap0.4), so the inflated gap (0.55–0.94) likely held growth shut, and the pool sat at 135 experts. Other brakes could also have contributed; this one was firing.plasticity.py). It was steering on a number dominated by untrained positions. This time it was already at its 1.0 cap.The bug was invisible on the released model, whose window had reached its 4,096 ceiling, where the two measurements are the same.
Change
The evaluator starts at the window the run resumes at (
context_now, clamped like the reading window), andev.contextis set to the current window before every evaluation: before training, each sample round, and after. Runs whose window equals the ceiling score exactly as before.Expect a one-time discontinuity: on runs below the ceiling, held-out drops by roughly 0.85 at the first evaluation after this change. Earlier
before/afternumbers and sample logs from such runs aren't comparable with later ones.Test plan
train.py readwith--held-out: before, after, savepytest: 22/22🤖 Generated with Claude Code