Repository navigation
Trainer: the call timeout and hard budget stop setup and finalize - #53
Open
Tong-bit-art wants to merge 1 commit into
Open
Tong-bit-art wants to merge 1 commit into
Tong-bit-art wants to merge 1 commit into
Conversation
The step call's context carried the Trainer's cancel event, so tools refused
further calls and started programs stopped once the call_timeout_min or the
hard budget was reached. The setup and finalize calls built their context
without the event, so the timeout and the budget set an event nothing read:
a setup or finalize that never returned hung the run past call_timeout_min
and past the budget, with no summary and nothing to resume from.
Both calls now build their context from the same cancel event the watchdog
sets, as _call's comment and the Context.cancel field already describe; the
docs' call_timeout_min guarantee ("each Evolver call (setup, a step,
finalize)") now holds for all three call kinds.
A regression test runs the Trainer in a subprocess: a setup that polls
ctx.cancel ends at call_timeout_min instead of hanging the run.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change
The step call's context carried the Trainer's cancel event, so its tools refused further calls and started programs stopped once
trainer.call_timeout_minor the hard budget was reached. The setup and finalize calls built their context without the event, so the timeout and the budget set an event nothing read: a setup or finalize that never returned hung the run pastcall_timeout_minand past the budget, with no summary and nothing to resume from (a restart re-enters the same call).Both calls now build their context from the same cancel event the watchdog sets, as
_call's comment and theContext.cancelfield already describe. No contract changes; the documentedcall_timeout_minguarantee — "the time, in minutes, the Trainer allows each Evolver call (setup, a step, finalize)" (docs/recipes/officeqa.md) — now holds for all three call kinds.Validation
tests/test_training.py::test_call_timeout_stops_the_setup: a setup that pollsctx.cancelends atcall_timeout_mininstead of hanging the run. The call runs in a subprocess, because while the timeout could not reach the context, this test would hang rather than fail.python -m pytest tests/test_training.py tests/test_spec.py tests/test_deploy.py: 38 passed. The 2test_spec.pyfailures (test_exports_are_learnable_and_restored_when_they_do_not_parse,test_adapter_md_may_be_removed) reproduce identically on unmodifiedmainin this environment (a sandboxed check subprocess cannot import jsonschema under this venv layout) and are unrelated to this change.hermit check,hermit train,hermit eval --adapter finalondocs/_examples/first_training/run.yaml): passes,mean_score: 1.0.Review