8000 Phase 6c: leakage-aware RL and evaluation export by Mnehmos · Pull Request #58 · Mnehmos/MathOS · GitHub
[go: up one dir, main page]
Skip to content

Phase 6c: leakage-aware RL and evaluation export - #58

Merged
Mnehmos merged 3 commits into
mainfrom
feat/57-rl-eval-export
Jul 22, 2026
Merged

Phase 6c: leakage-aware RL and evaluation export#58
Mnehmos merged 3 commits into
mainfrom
feat/57-rl-eval-export

Conversation

@Mnehmos
@Mnehmos Mnehmos commented Jul 22, 2026 •
edited
Loading
Copy link
Copy Markdown
Owner

Advances #57; keep the issue open through protected execution and independent artifact audit.

Summary

  • add closed Rust-owned rl_export_plan/1, rl_task/1, rl_leakage_report/1, and rl_export_manifest/1 contracts with committed offline schemas
  • bind cohort dates to signed publication receipts; require declared dependency/equivalence/source/certificate/proof/benchmark relations; add exact derived keys including normalized nonempty import manifests; union transitive components; reject cross-split or temporal leakage
  • permit private releases only as held_out_evaluation and require already-public licensed releases for train, validation, or public test
  • deterministically project formalization, fidelity-selection, counterexample, statement-repair, declaration-retrieval, and proof-generation tasks, plus eligible explanation/curriculum tasks; audit all twelve SPEC families with explicit skips
  • bind fidelity variants to exact claim versions and materialize every task's authority/fidelity snapshots as inspectable evidence
  • add database-independent release export-rl and verify-rl-export, trusted-hash verification, exact inventory/schema/task/evidence policy checks, and byte-identical reprojection
  • extend protected Pilot A to preview/build/copy/verify a 10-task private held-out export after the existing SQLite-hidden Lean release replay, with the retained plan naming the uploaded source release for independent audit
  • document the boundary and keep Comparator, four evidence-absent task projections, remaining pilots, and full 1.0 explicitly open

Local evidence

  • cargo fmt --check
  • cargo clippy --workspace --all-targets --all-features -- -D warnings
  • cargo test --workspace (167 library tests plus all integration suites)
  • python -m compileall -q src tests
  • python -m unittest discover -s tests -v (39 passed)
  • bash -n scripts/release-pilot-a.sh
  • actionlint 1.7.12
  • real frozen Pilot A offline preview/build/verify: manifest 91fa7a720c9ed0d530742c9c519cdb6f9944aa8545339d00e15e02f6846aac78, 30 members, 10 tasks, one held-out component, six emitted families, all tasks private, zero cross-claim fidelity variants, zero member mismatches, zero forbidden reasoning fields

Review focus

  • split-component union and receipt-derived temporal policy
  • private/public license and restriction preservation
  • exact task/evidence bindings and deterministic reprojection
  • protected workflow assertions and retained artifact scope

@coderabbitai
coderabbitai Bot commented Jul 22, 2026 •
edited
Loading
Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a deterministic, leakage-aware RL evaluation export pipeline with schemas, domain validation, CLI commands, offline verification, Pilot A publication integration, documentation, and tests.

Changes

RL evaluation export

Layer / File(s) Summary
RL export contracts and policy
.gitattributes, README.md, docs/decisions/*, docs/implementation/*, schemas/release/*, src/domain/*		 Defines RL plan, task, leakage-report, and manifest schemas; adds domain types and validation; documents component-locked splits, evidence policies, task families, and release status.
Deterministic RL projection
src/lib.rs, src/cli/mod.rs, src/rl_export.rs		 Adds export-rl and verify-rl-export, derives leakage components, projects task families and evidence, and publishes deterministic manifests and artifacts.
Offline integrity verification
src/rl_export.rs, tests/phase1_cli.rs		 Validates canonical JSON, schemas, inventories, hashes, bindings, split isolation, deterministic reprojection, filesystem safety, and CLI error behavior.
Pilot A publication integration
scripts/release-pilot-a.sh, .github/workflows/publication.yml		 Adds RL output handling, dry-run/build hash checks, verification, playtest metadata, and publication artifact retention.
Estimated code review effort: 5 (Critical) ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ReleaseCLI
  participant RLExporter
  participant FrozenReleases
  participant PublishedExport
  ReleaseCLI->>RLExporter: export-rl with plan and source root
  RLExporter->>FrozenReleases: load manifests, receipts, records, and evidence
  RLExporter->>RLExporter: derive leakage components and task projections
  RLExporter->>PublishedExport: publish manifest and export members
  ReleaseCLI->>RLExporter: verify-rl-export with expected manifest hash
  RLExporter->>PublishedExport: verify inventory and member hashes
  RLExporter->>FrozenReleases: reproject from independent plan and sources
  RLExporter-->>ReleaseCLI: return verification report
Loading

Possibly related issues

  • Issue 57 — The changes implement the tracked leakage-aware RL/evaluation export, verification, Pilot A, and audit scope.
  • Issue 50 — The export pipeline consumes frozen release bundles and builds on the existing offline replay and verification foundation.
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 13.13% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description includes a summary and evidence, but omits most required template sections and many required details. Expand it to match the template with all required sections: change class, trust boundary, TDD evidence, mathematical verification, provenance, risk/recovery, and review readiness.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and accurately summarizes the main change: a Phase 6c leakage-aware RL/evaluation export update.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/57-rl-eval-export

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Mnehmos Mnehmos mentioned this pull request Jul 22, 2026
Closed
6 tasks done
@Mnehmos
Mnehmos marked this pull request as ready for review July 22, 2026 03:32
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jul 22, 2026
View reviewed changes
@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1901c75e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/rl_export.rs Resolved
Comment thread src/rl_export.rs Resolved
@Mnehmos
Mnehmos merged commit 4b0a740 into main Jul 22, 2026
6 checks passed
This was referenced Jul 22, 2026
Merged
Open
@coderabbitai coderabbitai Bot mentioned this pull request Jul 22, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet
< 44EA div class="discussion-sidebar-item sidebar-progress-bar js-discussion-sidebar-item js-socket-channel js-updatable-content" data-channel="eyJjIjoicHVsbF9yZXF1ZXN0OjQxMDQ4NTAwODQiLCJ0IjoxNzk0NTI4NzIzfQ==--d16849b0b8c8c7c4c386505cd0b2b5fbd3ba872be54028704d978dbddb4b3e6f" data-gid="PR_kwDOTc7SCs70qwqk" data-url="/Mnehmos/MathOS/issues/58/show_partial?partial=issues%2Fsidebar%2Fshow%2Fmilestone" data-channel-event-name="milestone_updated" >

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Mnehmos
0