8000 Reset serving cache before RoPE rollover by GroveMinting · Pull Request #9 · volotat/mini-AGI · GitHub
[go: up one dir, main page]
Skip to content

Reset serving cache before RoPE rollover - #9

Open
GroveMinting wants to merge 1 commit into
volotat:mainfrom
GroveMinting:fix/serve-context-rollover
Open

Reset serving cache before RoPE rollover#9
GroveMinting wants to merge 1 commit into
volotat:mainfrom
GroveMinting:fix/serve-context-rollover

Conversation

@GroveMinting
@GroveMinting GroveMinting commented Sep 22, 2026
Copy link
Copy Markdown

Fixes #8.

What

  • Track serving cache positions explicitly during prompt prefill and generation.
  • Stop trimming already-rotated keys and rebasing the next query onto reused positions.
  • Release a full cache, then re-prefill the latest half-window from position zero.
  • Release the old cache before allocating its replacement so peak KV memory does not rise to roughly 1.5x.
  • Share prompt and rollover prefill through one helper.

Why

Retained KV entries keep the RoPE positions at which they were computed. Trimming the oldest entry and deriving pos_offset from the shorter cache assigned the next byte the same position as the retained final key, corrupting relative positions for the rest of generation. RecurCoder.generate() already avoids this by rebuilding a recent half-window.

Verification

python -m unittest discover -s tests -v

Three CPU tests pass. They verify:

  • monotonic offsets during chunked prefill;
  • exact retained tokens and generated continuation across repeated rollovers;
  • every forward remains within the rotary table;
  • the full cache is released before replacement allocation;
  • chunked prefill and a full RecurCoder forward produce matching final logits.

Also verified with compileall and git diff --check. This PR is independent of #7 and branches from the same upstream main commit.

This was referenced Sep 22, 2026
Open
Open
Open
@HenricHuntsman HenricHuntsman mentioned this pull request Sep 26, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Serving reuses RoPE positions after the KV cache fills

1 participant

@GroveMinting
0