8000 Publish reproducible benchmark results and downloadable evidence by rafatahmed · Pull Request #43 · rafatahmed/Flahax · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,6 @@ tests/fixtures/phreeqc/**/*.sel -text whitespace=-blank-at-eol,-blank-at-eof
tests/fixtures/phreeqc/**/*.screen -text whitespace=-blank-at-eol,-blank-at-eof
tests/fixtures/phreeqc/**/error.inp -text whitespace=-blank-at-eol,-blank-at-eof
tests/fixtures/phreeqc/**/*.json -text whitespace=cr-at-eol

# Published evidence downloads must match their archived bytes and checksums.
docs/site/_static/benchmarks/** -text whitespace=cr-at-eol
9 changes: 6 additions & 3 deletions .github/workflows/benchmark.yml
8000
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,10 @@ jobs:
python-version: "3.13"
- name: Install benchmark tooling
id: install
run: python -m pip install -e . -r benchmarks/requirements.txt
run: |
python -m pip install -e . -r benchmarks/requirements.txt
mkdir -p build/benchmark-v1
python -m pip freeze --all > build/benchmark-v1/environment.txt
- name: Measurement semantics and correctness scorecards
run: |
python -m unittest tests.test_benchmark_baseline
Expand All @@ -28,9 +31,9 @@ jobs:
if: ${{ !cancelled() && steps.install.outcome == 'success' }}
shell: bash
run: |
python -m asv machine --yes --machine github-linux
python -m asv machine --yes
python -m asv check --python=same
python -m asv run --python=same --machine github-linux --set-commit-hash "$GITHUB_SHA" --record-samples --show-stderr 2>&1 | tee build/benchmark-v1/asv.log
python -m asv run --python=same --set-commit-hash "$GITHUB_SHA" --record-samples --show-stderr 2>&1 | tee build/benchmark-v1/asv.log
- name: Record provenance and sample summary
if: always()
run: python -m tools.benchmark_baseline --metadata-only --compare-inputs build/benchmark-v1/before.json --asv-results build/asv/results --output build/benchmark-v1/after.json
Expand Down
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ include asv.conf.json
recursive-include benchmarks *.py *.txt
recursive-include docs *.md *.jpg *.png *.pdf
recursive-include docs/site *.csv
recursive-include docs/site/_static/benchmarks *.json *.svg *.zip *.sha256
exclude docs/sources/04_Fertilizer.pdf
exclude docs/sources/165357805552.pdf
exclude docs/sources/215057776303.pdf
Expand All @@ -24,6 +25,7 @@ recursive-include src/flahax/data *.md
recursive-include tests *.py
recursive-include tests/fixtures *.json *.pqi *.out *.sel *.screen *.dat *.md *.inp
recursive-include tools *.py *.ps1
include tools/requirements-benchmark-report.txt
recursive-include examples *.py *.md
prune examples/tutorial
global-exclude __pycache__
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ The [Benchmark v1 protocol](docs/site/evidence/benchmark-v1.md) separates
formulation fit, numerical chemistry, runtime/memory measurements, and physical
validation. Its reproducible reports include input hashes and failed cases;
timings are informational, and physical validation remains separate.
See the [measured results and downloadable evidence](docs/site/evidence/benchmark-results.md)
for tables, charts, provenance, and retained limitations.

- 153 automated tests, including live replay of every catalogue product,
phase and acid case against pinned PHREEQC 3.8.6 (USGS).
Expand Down
2 changes: 1 addition & 1 deletion asv.conf.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"project": "FlahaX",
"project_url": "https://github.com/rafatahmed/Flahax",
"repo": ".",
"branches": ["main"],
"branches": ["HEAD"],
"environment_type": "virtualenv",
"pythons": ["3.13"],
"matrix": {"numpy": ["2.5.4"], "scipy": ["1.18.1"]},
Expand Down
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ Readers: the documentation is at <https://flahax.readthedocs.io/>. This index is
## Scientific model and evidence

- [Benchmark v1 specification and reproduction](site/evidence/benchmark-v1.md): separate correctness and performance scorecards, fixed inputs, provenance, and retained failures.
- [Benchmark results and downloadable evidence](site/evidence/benchmark-results.md): a clean-checkout snapshot with tables, charts, raw records, and retained limitations.
- [Formulation method](site/models/method.md)
- [Current mixed-equilibrium model](site/models/fertilizer-equilibrium-model.md)
- [Coverage summary](site/models/full-fertilizer-chemistry.md) and [exact reaction/phase matrix](site/models/chemistry-coverage-matrix.md)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
43386a611b64cf906158205e15973b9e15f1f56fcb466fa3933d1522f30b58ec README.txt
f99f6abdfc8d43f9e2f31ac155ae0491cf3166164c7f62c21baaf8da0f2326c0 asv/benchmarks.json
a9196208a86a1ce31c4e0b149a2523556eee998b8d36d2d402031adc3bff0698 asv/runnervm8k3b6/2340a19c-existing-py_opt_hostedtoolcache_Python_3.13.16_x64_bin_python.json
fe3ddacd9e176673d737d2bd1195f2acd98cb28cd928cb32a6cc9753d243b079 asv/runnervm8k3b6/machine.json
028b30ac8388daac5c020d7fc2660a4ea97f20f8e8bff0b66103cc467d458d92 baseline.json
96624b259490e98e7d5fd78d29ef9c8c15be74744be0a796f189c19be12fa757 capture/after.json
3fb215f15eb1f6ec6258db749f24561fbd528f628ad45f05c0fab3882b4bb673 capture/asv.log
db4fa48113eb85c364d20bdb6aa76434fc537aef3e527cb6d8e73fed736c7159 capture/before.json
c7638a73b7d68d8e4004ebab3a5621128b790c561a47f9d5528a2c93b105f182 capture/environment.txt
e01b15a325ada20871d70c62403233ca82c9bb0b3f85f538747c48189c7e7b4c capture/hosted-ci.json
932e98488abb40677de970159b758e0560be3bc6b092aa663ae902334c34a7bd capture/hosted-workflow.log
84392ca9f0d1ec073b8d64acea2bebfbec4901827e5b99b283ce49775adc81a5 capture/manual-workflow.json
6dd88b1c7fd5220fc214d6c951f650c88a11feb42b786eeb6dd647b16f4e113f capture/publication-notes.txt
f3df362be48c0166a1ebe5e66622d2184c944ea9e536be55db7576aba028b8b8 configuration/.github/workflows/benchmark.yml
091019434d1e0e1f5e789595503f74a856a9eb634fd8260e744fd471e648798f configuration/asv.conf.json
f61b1f8a9635d7ce95943d3644805a5460cc38defcaea2f260a6b0df094d098d configuration/benchmarks/requirements.txt
3d82e8562062533ce76a384583518dec90e0eea561131d00fb683e78d9edcb97 formulation.csv
032a618ee4e176002155095c739ea2504e1b7496e88c0b8a8c64b2c6fab4aa35 formulation.png
b790f4c4d5b44034519288d7b9e525408ddb8a86e4911d5461d63901fff33f55 formulation.svg
dad031eea253a6467e1a6a44faa0f3d940c2fc1a5c31ea55795fe421492128f8 publish_benchmark_report.py
886ec157e10373edcae9eb4c3506b3b48c412c6feeb1cffe96c613fd9cb48ef8 renderer-environment.json
3098cfcf748422ba3c6f4333d893ff67bcf4401dd106b46b6b704e4f4dd25819 requirements-benchmark-report.txt
46d688b858daa6fabd19577b822cf41966a217f25b679a3dd62dfa7eb7002ab8 timings.csv
cb3db5a01b9f862820f5753823bcd80bba644d39aca921b69dc511710ddd8811 timings.png
84957a9923ee3866d5500e24255568da9fe25fd840ccf728760480757abc4e4a timings.svg
Loading
Loading
0