Repository navigation
GradSNR: count a missing grad as zeros instead of dropping it - #20
Open
dreddnafious wants to merge 1 commit into
Open
dreddnafious wants to merge 1 commit into
dreddnafious wants to merge 1 commit into
Conversation
…t#19) observe() concatenated only the parameters that had a gradient, so the flattened vector changed length whenever one sat a step out and the running mean no longer matched it. Recurrence depth sampling makes that routine: a step sampled at depth 1 forces the halt and leaves the halting head without a gradient (d_model + 1 elements - the 129 in volotat#19 at d_model 128, 257 at d_model 256). A missing gradient is a zero gradient for that step, so it is counted as zeros; the meter keeps its history rather than resetting. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Sep 23, 2026
Open
10xCrazyHorse
added a commit
to yaaa-cool/mini-AGI
that referenced
this pull request
Oct 3, 2026
observe concatenated only the parameters that had a gradient, so the flattened vector changed length whenever one sat a step out, and the running mean no longer matched it. A step sampled at recurrence depth 1 forces the halt, and the halting head (d_model + 1 elements) gets no gradient. A missing gradient is a zero gradient for that step, so it is counted as zeros and the meter keeps its history instead of resetting. The optim.py change is identical to upstream PR volotat#20 by @dreddnafious (reported by @nekomario28), so it merges cleanly if volotat#20 lands. Adds tests/test_bug19_gradsnr.py (CPU, tiny model).
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #19 (reported by @nekomario28).
GradSNR.observeconcatenated only the parameters that had a gradient, so the flattened vector changed length whenever one sat a step out, and the running mean no longer matched it. Recurrence-depth sampling makes this routine: a depth-1 step forces the halt, and the halting head (d_model + 1 elements) gets no gradient.A missing gradient is a zero gradient for that step, so it's now counted as zeros. The meter keeps its history instead of resetting. It still returns
Nonewhen no parameter has a gradient.Tested: a regression script (intermittently gradless parameter; constant gradient still reads ~1.0) fails on
mainand passes here. The same change ran ~8 GPU-hours oftrain.py readat a small scale without issue.One file, +10/-3.
🤖 Generated with Claude Code