A set of tools to record the conditions (code, parameters, ...) and agregate the results of many evaluation runs.
- Reproducability (see tensorflow#591)
- Aid sharing
- Communicate exactly (diffs, commands, ...) what was done
- Store all data in a common repository with common naming scheme
- Be easy to extend piecewise as needed
- Be backwards compatible as often as possible
- Guide us towards gating (see tensorflow#570_
- Results table / Evaluation (tensorflow#591)
- Uses bigtable tag #590 to name a comparison
- Use launch_eval.py and record command (somewhere)
- Directory structure
- gs://minigo-pub/experiments/eval/<experiment-tag>/
- sgf/eval/
- YYYY-MM-DD/
- @TS-...-<model_1>-...-<model_2>...sgf (e.g. 1540317107-000011-malabar-000010-defence-200.sgf)
- YYYY-MM-DD/
- results.html (autogenerated)
- command_@TS (e.g command_1544086567)
- Command line invocation of
launch_eval.py
- Command line invocation of
- metadata (json?, contents TBD)
- [optional] command_flags
- [future] <shorttag>_ringmaster.ctl (see tensorflow#544)
- [future] branch (github branch where code can be found)
- [future] diff.patch
- sgf/eval/
- gs://minigo-pub/experiments/eval/<experiment-tag>/