8000 minigo/cluster/eval_server/README.md at master · jm16/minigo · GitHub
[go: up one dir, main page]

Skip to content
< 8000 /react-partial>

Latest commit

 

History

History
33 lines (27 loc) · 1.46 KB

README.md

File metadata and controls

33 lines (27 loc) · 1.46 KB

Minigo Evaluation Logger

A set of tools to record the conditions (code, parameters, ...) and agregate the results of many evaluation runs.

Goals

  1. Reproducability (see tensorflow#591)
  2. Aid sharing
    1. Communicate exactly (diffs, commands, ...) what was done
  3. Store all data in a common repository with common naming scheme
  4. Be easy to extend piecewise as needed
    1. Be backwards compatible as often as possible
  5. Guide us towards gating (see tensorflow#570_

Methodology

  • Results table / Evaluation (tensorflow#591)
    • Uses bigtable tag #590 to name a comparison
    • Use launch_eval.py and record command (somewhere)
    • Directory structure
      • gs://minigo-pub/experiments/eval/<experiment-tag>/
        • sgf/eval/
          • YYYY-MM-DD/
            • @TS-...-<model_1>-...-<model_2>...sgf (e.g. 1540317107-000011-malabar-000010-defence-200.sgf)
        • results.html (autogenerated)
        • command_@TS (e.g command_1544086567)
          • Command line invocation of launch_eval.py
        • metadata (json?, contents TBD)
        • [optional] command_flags
        • [future] <shorttag>_ringmaster.ctl (see tensorflow#544)
        • [future] branch (github branch where code can be found)
        • [future] diff.patch
0