8000 Build a dedicated self-knowledge corpus lane by GroveMinting · Pull Request #11 · volotat/mini-AGI · GitHub
[go: up one dir, main page]

Skip to content

Build a dedicated self-knowledge corpus lane - #11

Open
GroveMinting wants to merge 1 commit into
volotat:mainfrom
GroveMinting:fix/self-knowledge-corpus
Open

GroveMinting wants to merge 1 commit into
volotat:mainfrom
GroveMinting:fix/self-knowledge-corpus

Conversation

@GroveMinting
Copy link
Copy Markdown

Fixes #10.

What

  • Add chat --self-only generation backed strictly by self_knowledge.yaml.
  • Skip Python harvesting, arithmetic, and generic chat mixing in that mode.
  • Write generated binaries to data_self_knowledge_char by default.
  • Expand that source into data/{train,val}/self-knowledge.
  • Wire corpora all to the new source and make --limit bound its conversation count.
  • Treat nonzero builder return codes as failures even when stale output files exist.
  • Close meta.json explicitly and record the generation mode.

Why

The existing self-knowledge builder called the generic chat generator and expanded the chat lane. Since the expander had no self-knowledge source, data/train/self-knowledge was never created, corpora all reported failure, and serving could not load its documented prime.

Verification

python -m unittest discover -s tests -v

Five tests pass. They cover:

  • YAML-only generation without code harvesting;
  • binary train/validation streams expanded into the dedicated lane;
  • builder source/destination and --limit wiring;
  • explicit failure when YAML pairs are unavailable;
  • failed forced rebuilds not being hidden by stale files;
  • unchanged default mixed-chat generation.

Also verified with compileall and git diff --check. The changed files are independent of open PRs #7 and #9.

@dreddnafious
Copy link
Copy Markdown

One more thing this PR fixes, which may be worth noting: on main, the Hermes held-out files are written to data/val/chat, and then the self-knowledge step's expand --only chat wipes that folder. So a fresh python -m corpora all ends up with no Hermes held-out set at all. Since this PR expands self-knowledge into its own lane, the wipe goes away.

One thing it doesn't cover: the README scores chat and chat_hermes as separate held-out subjects, but the builder puts Hermes under train/chat/hermes and val/chat. That merges it into the chat lane (lanes are top-level folders). Writing Hermes to data/{train,val}/chat_hermes would make a fresh build match the README's eight subjects. For a replication I restored it with corpora fetch ... --hold 400 --held-out data/val/chat_hermes, which reproduces the same first-400 records.

Maxritz pushed a commit to Maxritz/mini-AGI-cpp that referenced this pull request Sep 25, 2026
…weep (case 2b)

The d_lam path was implemented but missing the 1/M batch mean, making
every halting gradient M times too large (8x at production M=8).
Verified via a cached-lam finite difference in double precision that
sweeps M={1,2,3}: rel error was exactly 1/M with the defect and
passed with the fix.

- src/backward.cpp: fold inv_M into step_ce so d_lam and dc_prev scale correctly
- src/backward.hpp: publish d_lam_per_step via optional out-param
- tests/test_backward.cpp: case 2b check_halting_scaling + halting_identity;
  also fixed a latent exp-for-log typo in ce_of that had masked an entire defect turn
- tests/test_pool_backward.cpp: covers pooled MLP in isolation
- src/paged.*: assert_slots_valid on mutable_w* to prevent pre-begin_segment writes
- src/recur.hpp: begin_segment() marked REQUIRED
- STATUS.md: gap volotat#11 CLOSED, gap volotat#12 CLOSED, ce_of typo recorded as bug volotat#5

Test: 8/8 ctest pass; defect reintroduction fails 6 checks, fix passes all.
Train smoke: 3 steps, loss 5.5458 (forward unchanged).
Maxritz pushed a commit to Maxritz/mini-AGI-cpp that referenced this pull request Sep 25, 2026
Maxritz pushed a commit to Maxritz/mini-AGI-cpp that referenced this pull request Sep 25, 2026
Maxritz pushed a commit to Maxritz/mini-AGI-cpp that referenced this pull request Sep 25, 2026
Maxritz pushed a commit to Maxritz/mini-AGI-cpp that referenced this pull request Sep 25, 2026
10xCrazyHorse added a commit to yaaa-cool/mini-AGI that referenced this pull request Oct 3, 2026
corpora all declared data/train/self-knowledge but build_self_knowledge ran
the generic chat generator and expanded the chat lane, so the folder that
serve.py primes from was never created. Add chat --self-only (YAML-backed
turns only, written to data_self_knowledge_char), expand it into
data/{train,val}/self-knowledge, bound it with --limit, and count a nonzero
builder return code as a failure even when stale files exist.

Same change as volotat#11, with its test as
tests/test_self_knowledge_lane.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Corpus build never creates the self-knowledge lane

2 participants

0