git pull --ff-only origin mainUse Python 3.11 and install dependencies:
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txtevaluation/: EX, R-VES, Soft-F1 evaluation scriptsllm/: LLM inference and prompt-related codelive_sql_bench_sqlite/: LiveSQLBench SQLite evaluation pipelinefinetuning/: data prep, inference, and finetuning helpers
- Edit
evaluation/run_evaluation.shand set:
predicted_sql_pathto your prediction filesql_dialect(for exampleSQLite)
- Run:
cd evaluation
sh run_evaluation.shThe script calls evaluation/evaluation_ex.py and writes results to eval_result/.
The repo includes single-instance evaluation code:
- Script:
live_sql_bench_sqlite/evaluation/single_instance_eval_sqlite.py
Expected usage:
python3 live_sql_bench_sqlite/evaluation/single_instance_eval_sqlite.py \
--jsonl_file <single_instance.jsonl> \
--output_file <result.json> \
--mode pred \
--logging trueNotes:
- The script reads the first record in
--jsonl_fileas one instance. - It supports
--mode predand--mode gold. - By default it uses a hardcoded SQLite DB path (
/Volumes/SN770/...), unlessEPHEMERAL_DB_PATHis set. - You may need to adjust database path logic in this script for your local machine.
- Keep your prediction files in a separate folder (for example
sql_result/). - Keep your evaluation outputs in
eval_result/. - Include this README and your command history in your homework submission.