Repository navigation
Corpus: OpenHermes held out on its own; self-knowledge check finds its files - #5
Merged
Merged
Conversation
…s files `python -m corpora all` had two path bugs in the chat lanes: - OpenHermes wrote its held-out set into data/val/chat, and the generated chat lane (primal chat + self-knowledge) runs `expand --only chat` afterwards, which clears the .txt files in data/val/chat - silently deleting OpenHermes' 400 held-out files on every full build. It now holds out into data/val/chat_hermes, the separate held-out subject the README's results already report. Its training data stays in data/train/chat/hermes. - The self-knowledge lane was checked at data/train/self-knowledge, a folder nothing writes, so every build reported "self-knowledge produced nothing" and exited 1 although the lane had been written to data/train/chat. It is checked by glob, data/train/chat/chat-*.txt - not by the folder, which holds hermes/ too and would read as built before the lane ever was. serve.py's priming read the same missing folder, so it served unprimed; it reads data/train/chat now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two path bugs in the chat lanes of
python -m corpora all, found while building the full corpus on Windows.data/val/chat. The generated chat lane (primal chat plus the self-knowledge turns) runs afterwards withexpand --only chat, which clears the.txtfiles indata/{train,val}/chat, and that took OpenHermes' 400 held-out files with it.data/val/chat_hermes, the separate held-out subject the README's results already report (chat_hermes 0.924).data/train/chat/hermes, inside the chat subject as before.data/train/self-knowledge, a folder nothing writes, so every full build ended withlanes that produced nothing: self-knowledgeand exit code 1, even though the lane had been written todata/train/chat.data/train/chat/chat-*.txt. Checking the folder itself wouldn't work: it also holdshermes/, so it would read as built before the lane ever was._files()replaces the directory walk in both the skip check and the summary.serve.py's priming read the same missing folder, so it always served with no priming. It readsdata/train/chatnow.The first passage it picks is a code-explanation exchange rather than a question about the model itself. The generated lane mixes both kinds and doesn't name them apart the way upstream's
self-*files did. That's left as is.Test plan
chat-*.txt)python -m corpora all --only chat --forcewrites 400 held-out files todata/val/chat_hermes, anddata/val/chatkeeps its 22data/valnow has the eight subjects the README reports: arithmetic, chat, chat_hermes, chess, code, reasoning, stories, wikipediaserve.load_prime(1024)returns a 566-character balanced passage, where before it returned nothingpytest: 22/22🤖 Generated with Claude Code