8000 Fix benchmark publication audit gaps by rafatahmed · Pull Request #50 · rafatahmed/Flahax · GitHub
[go: up one dir, main page]
Skip to content

Fix benchmark publication audit gaps #50

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rafatahmed merged 3 commits into from
Oct 20, 2026
Merged

Fix benchmark publication audit gaps #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
960f66b
Clarify benchmark results pages and name the largest misses
rafatahmed Oct 20, 2026
c256039
Use absolute links for README images and benchmark pages
rafatahmed Oct 20, 2026
8dd3c97
Remove internal roadmap codes from public docs pages
rafatahmed Oct 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# FlahaX

<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/site/_static/brand/v2/flahax-banner-white.png">
<img src="docs/site/_static/brand/v2/flahax-banner-color.png" alt="FlahaX — water and nutrients become chemistry calculations and traceable plans, with units, model identifiers and sources." width="100%">
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rafatahmed/Flahax/main/docs/site/_static/brand/v2/flahax-banner-white.png">
<img src="https://raw.githubusercontent.com/rafatahmed/Flahax/main/docs/site/_static/brand/v2/flahax-banner-color.png" alt="FlahaX — water and nutrients become chemistry calculations and traceable plans, with units, model identifiers and sources." width="100%">
</picture>

**Plant-nutrient chemistry engine in Python: fertilizer formulation, aqueous
Expand Down Expand Up @@ -100,11 +100,11 @@ plan; they never operate pumps, valves or other hardware.

## Verification

The [Benchmark v1 protocol](docs/site/evidence/benchmark-v1.md) separates
The [Benchmark v1 protocol](https://flahax.readthedocs.io/en/latest/evidence/benchmark-v1.html) separates
formulation fit, numerical chemistry, runtime/memory measurements, and physical
validation. Its reproducible reports include input hashes and failed cases;
timings are informational, and physical validation remains separate.
See the [measured results and downloadable evidence](docs/site/evidence/benchmark-results.md)
See the [measured results and downloadable evidence](https://flahax.readthedocs.io/en/latest/evidence/benchmark-results.html)
for tables, charts, provenance, and retained limitations.

- 153 automated tests, including live replay of every catalogue product,
Expand Down
2 changes: 1 addition & 1 deletion docs/chemistry-coverage-matrix.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# P2/P3 chemistry coverage matrix
# Chemistry coverage matrix

This page moved to the FlahaX documentation: <https://flahax.readthedocs.io/>, section "Models and chemistry".
6 changes: 3 additions & 3 deletions docs/site/api/package-verification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ The canonical manual is `src/flahax/data/USER_GUIDE.md`. It is bundled as a pack

To retain only artifacts that passed clean-install checks, use `python tools/verify_distribution.py --artifacts-dir dist/<version>`. The destination must be new or empty; existing release files are never overwritten. This command builds and tests locally; it does not tag, upload or publish. Afterward, run `python -m twine check --strict dist/<version>/*` in a development environment with `twine` installed. The build and metadata tools are not runtime dependencies. Releases themselves are built and uploaded by the [hardened release workflow](https://github.com/rafatahmed/Flahax/blob/main/docs/publishing.md).

## P6.6: installed distributions
## Installed distributions

Run `python -m pip install build` in a development virtual environment, then `python tools/verify_distribution.py`. The tool builds wheel and sdist (wheel from sdist), installs each non-editably in a separate temporary virtual environment, runs from outside the checkout with `-I` and without `PYTHONPATH`, and asserts the imported package is inside that environment. It verifies zero runtime dependencies, 28 catalogue products, the exact chemistry resource hash, catalogue solve, nitric target, equilibrium delivery adapter, and both `python -m flahax` and `flahax` CLI entry points. A JSON report containing distribution hashes and observed numerical values is written to `build/distribution-verification.json`. Nothing is published.

The source distribution now includes offline fixture inputs/outputs, extensions and reproduction tools as well as tests; the wheel includes runtime Python and JSON resources only. Neither contains PHREEQC binaries. Python 3.11–3.13 CI runs source regression and clean distribution smoke checks. Local Python 3.14 testing is additional evidence, not a substitute for that matrix.

## P6.7: mandatory live reference validation
## Mandatory live reference validation

`tools/provision_phreeqc.ps1` uses the [official USGS Windows 3.8.6-17100 installer](https://water.usgs.gov/water-resources/software/PHREEQC/). It verifies installer SHA-256 before administrative extraction into TEMP, then verifies executable and database SHA-256. An existing installation is reused only after hash verification. No PATH, base database or global installation is changed. The Windows CI job sets `FLAHAX_PHREEQC_ROOT` and `FLAHAX_REQUIRE_PHREEQC=1`; missing tools are an error, not a skipped success.

Expand All @@ -25,7 +25,7 @@ python -m unittest tests.test_product_phreeqc -v

Ordinary tests may still skip live replay when PHREEQC is unavailable, while all checked-in golden regressions remain mandatory. CI uploads live test logs and distribution reports. Remote CI cannot be claimed passed while GitHub's account billing lock prevents jobs from starting.

## P5.5: explicit equilibrium delivery API
## Explicit equilibrium delivery API

`plan_equilibrium_delivery` returns **EquilibriumPhPlan**, distinct from the measured-curve **PhPlan**. Both are accepted by `compose_delivery_plan`; no existing measured-curve signature or CLI recommendation contract changes. `plan_nitric_acid_target` and `plan_catalogue_nitric_target` return the numerical **AcidResult**; callers requiring reagent volume and delivery composition use the new adapter. Equal initial and target pH now returns exactly zero acid.

Expand Down
132 changes: 77 additions & 55 deletions docs/site/evidence/benchmark-results-windows.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,24 @@
# Benchmark results: Windows
# Benchmark results: Windows (local supplement)

**Measured snapshot: 2026-10-20-windows-6373d23.** This page reports one clean-checkout development
baseline, including failures. It is not a release certification, a comparison
against another product, or physical validation.

**Local supplementary capture.** This was measured on a developer machine, not a hosted runner.
Its timings reflect that machine's power and thermal state; use them only alongside the hosted capture.

```{warning}
The numerical scorecard is not fully passing: 6/7 controlled
NNLS cases pass. 2 timing workloads have no completed measurement.
The [manual GitHub benchmark workflow](https://github.com/rafatahmed/Flahax/actions/runs/39403201151) completed with conclusion
**failure**. A completed capture is not necessarily a passing
scientific scorecard; failed gates and timeouts are retained.
scientific scorecard; failed gates and timeouts are retained. Passing pull-request
CI checks are separate evidence and do not mean this scorecard passed.
```

Read the [protocol and reproduction instructions](benchmark-v1.md) before comparing results.

[Results from the other recorded environment](benchmark-results.md). Treat environments separately; this is not a controlled OS comparison.
[Hosted Linux baseline](benchmark-results.md). Treat environments separately; this is not a controlled OS comparison.

Historical Horticalc statistics in the evidence JSON are **not newly measured** and
are deliberately excluded from the charts on this page.
Expand Down Expand Up @@ -42,6 +46,22 @@ Percentage statistics exclude zero targets; absolute residual metrics retain the
There were 0 formulation exceptions; their
element rows are excluded, while attempted recipes remain in the recipe denominator.

### Where targets were missed

The largest miss is Si in Rice (Douglas Peckenpaugh), zero nutrients (`cmbpfud3e0092q8z9vq8gw25x:zero_nutrients`): target 100 mg/L, delivered 0 mg/L (-100.0%).

| Nutrient | Failing targets |
|---|---:|
| Ca | 22/52 |
| N_NO3 | 17/52 |
| Mg | 12/52 |
| S | 12/52 |
| K | 6/52 |
| N_NH4 | 2/10 |
| Si | 2/2 |

Per-case causes are tracked in [recipe-fit analysis #39](https://github.com/rafatahmed/Flahax/issues/39).

## Timing and memory

![Completed timing samples, medians and ranges on a logarithmic millisecond axis](../_static/benchmarks/2026-10-20-windows-6373d23/timings.svg)
Expand Down Expand Up @@ -76,6 +96,8 @@ does not establish a cold filesystem cache.

Peak memory is total process peak RSS, including the interpreter and setup,
not the incremental allocation of the solver. Decimal MB are used.
Only 6 workloads have a memory
benchmark; the other workloads are timed only.

| Workload / parameter | Peak process memory (MB) |
|---|---:|
Expand Down Expand Up @@ -192,56 +214,56 @@ redistributed files; the archive explains the path-only redaction.
:::{dropdown} All recipe cases
| Case | Targets passing | Whole recipe |
|---|---:|---|
| cmbpfud0y006wq8z9ghn6t86g:zero_nutrients | 11/12 | Miss |
| cmbpfud0y006wq8z9ghn6t86g:calcium_20_mg_l | 11/12 | Miss |
| cmbpfud9f00eaq8z92ve3zic0:zero_nutrients | 10/12 | Miss |
| cmbpfud9f00eaq8z92ve3zic0:calcium_20_mg_l | 11/12 | Miss |
| cmbpfucsg0002q8z9a5q0u7nu:zero_nutrients | 12/12 | Pass |
| cmbpfucsg0002q8z9a5q0u7nu:calcium_20_mg_l | 12/12 | Pass |
| cmbpfud6y00c1q8z9emlc27x2:zero_nutrients | 12/12 | Pass |
| cmbpfud6y00c1q8z9emlc27x2:calcium_20_mg_l | 12/12 | Pass |
| cmbpfudcu00haq8z9xwjoq1zt:zero_nutrients | 11/13 | Miss |
| cmbpfudcu00haq8z9xwjoq1zt:calcium_20_mg_l | 13/13 | Pass |
| cmbpfud7r00crq8z9qc0abr6h:zero_nutrients | 7/12 | Miss |
| cmbpfud7r00crq8z9qc0abr6h:calcium_20_mg_l | 7/12 | Miss |
| cmbpfudbu00giq8z9jquzw746:zero_nutrients | 9/13 | Miss |
| cmbpfudbu00giq8z9jquzw746:calcium_20_mg_l | 13/13 | Pass |
| cmbpfucvl002kq8z9in7tpn5y:zero_nutrients | 12/12 | Pass |
| cmbpfucvl002kq8z9in7tpn5y:calcium_20_mg_l | 12/12 | Pass |
| cmbpfucun001uq8z9dgqeihsj:zero_nutrients | 12/12 | Pass |
| cmbpfucun001uq8z9dgqeihsj:calcium_20_mg_l | 12/12 | Pass |
| cmbpfud8k00dhq8z9yz59ziwb:zero_nutrients | 13/13 | Pass |
| cmbpfud8k00dhq8z9yz59ziwb:calcium_20_mg_l | 13/13 | Pass |
| cmbpfuddo00i2q8z9nldrxm9w:zero_nutrients | 13/13 | Pass |
| cmbpfuddo00i2q8z9nldrxm9w:calcium_20_mg_l | 13/13 | Pass |
| cmbpfud2n008cq8z99v63xpo5:zero_nutrients | 12/12 | Pass |
| cmbpfud2n008cq8z99v63xpo5:calcium_20_mg_l | 12/12 | Pass |
| cmbpfudb100fqq8z9wcljgmyi:zero_nutrients | 7/13 | Miss |
| cmbpfudb100fqq8z9wcljgmyi:calcium_20_mg_l | 7/13 | Miss |
| cmbpfud3e0092q8z9vq8gw25x:zero_nutrients | 11/13 | Miss |
| cmbpfud3e0092q8z9vq8gw25x:calcium_20_mg_l | 11/13 | Miss |
| cmbpfuctq0014q8z9li5oo75d:zero_nutrients | 7/12 | Miss |
| cmbpfuctq0014q8z9li5oo75d:calcium_20_mg_l | 7/12 | Miss |
| cmbpfudel00iuq8z9l2wiwsfy:zero_nutrients | 12/12 | Pass |
| cmbpfudel00iuq8z9l2wiwsfy:calcium_20_mg_l | 12/12 | Pass |
| cmbpfuda900f0q8z9hi9hu2gu:zero_nutrients | 11/12 | Miss |
| cmbpfuda900f0q8z9hi9hu2gu:calcium_20_mg_l | 12/12 | Pass |
| cmbpfud6400bbq8z9gwrj4cw7:zero_nutrients | 12/12 | Pass |
| cmbpfud6400bbq8z9gwrj4cw7:calcium_20_mg_l | 12/12 | Pass |
| cmbpfud5900alq8z907fk2r92:zero_nutrients | 12/12 | Pass |
| cmbpfud5900alq8z907fk2r92:calcium_20_mg_l | 12/12 | Pass |
| cmbpfud4a009vq8z953niqg47:zero_nutrients | 10/12 | Miss |
| cmbpfud4a009vq8z953niqg47:calcium_20_mg_l | 10/12 | Miss |
| cmbpfucxi0040q8z94o3m2bps:zero_nutrients | 12/12 | Pass |
| cmbpfucxi0040q8z94o3m2bps:calcium_20_mg_l | 12/12 | Pass |
| cmbpfucz7005gq8z90ev9h6ao:zero_nutrients | 8/12 | Miss |
| cmbpfucz7005gq8z90ev9h6ao:calcium_20_mg_l | 8/12 | Miss |
| cmbpfucyc004qq8z9mdajrg3w:zero_nutrients | 8/12 | Miss |
| cmbpfucyc004qq8z9mdajrg3w:calcium_20_mg_l | 12/12 | Pass |
| cmbpfud030066q8z9fvdyru1m:zero_nutrients | 8/12 | Miss |
| cmbpfud030066q8z9fvdyru1m:calcium_20_mg_l | 8/12 | Miss |
| cmbpfucwh003aq8z9vmh9kner:zero_nutrients | 12/12 | Pass |
| cmbpfucwh003aq8z9vmh9kner:calcium_20_mg_l | 12/12 | Pass |
| cmbpfud1t007mq8z9w16hx6hw:zero_nutrients | 12/12 | Pass |
| cmbpfud1t007mq8z9w16hx6hw:calcium_20_mg_l | 11/12 | Miss |
| Chilli (maximumyield), zero nutrients (`cmbpfud0y006wq8z9ghn6t86g:zero_nutrients`) | 11/12 | Miss |
| Chilli (maximumyield), calcium 20 mg l (`cmbpfud0y006wq8z9ghn6t86g:calcium_20_mg_l`) | 11/12 | Miss |
| Cucumber (Howard Resh), zero nutrients (`cmbpfud9f00eaq8z92ve3zic0:zero_nutrients`) | 10/12 | Miss |
| Cucumber (Howard Resh), calcium 20 mg l (`cmbpfud9f00eaq8z92ve3zic0:calcium_20_mg_l`) | 11/12 | Miss |
| DEFAULT, zero nutrients (`cmbpfucsg0002q8z9a5q0u7nu:zero_nutrients`) | 12/12 | Pass |
| DEFAULT, calcium 20 mg l (`cmbpfucsg0002q8z9a5q0u7nu:calcium_20_mg_l`) | 12/12 | Pass |
| Generic Bloom (maximumyield), zero nutrients (`cmbpfud6y00c1q8z9emlc27x2:zero_nutrients`) | 12/12 | Pass |
| Generic Bloom (maximumyield), calcium 20 mg l (`cmbpfud6y00c1q8z9emlc27x2:calcium_20_mg_l`) | 12/12 | Pass |
| Generic Dry Season (Howard Resh), zero nutrients (`cmbpfudcu00haq8z9xwjoq1zt:zero_nutrients`) | 11/13 | Miss |
| Generic Dry Season (Howard Resh), calcium 20 mg l (`cmbpfudcu00haq8z9xwjoq1zt:calcium_20_mg_l`) | 13/13 | Pass |
| Generic Grow (maximumyield), zero nutrients (`cmbpfud7r00crq8z9qc0abr6h:zero_nutrients`) | 7/12 | Miss |
| Generic Grow (maximumyield), calcium 20 mg l (`cmbpfud7r00crq8z9qc0abr6h:calcium_20_mg_l`) | 7/12 | Miss |
| Generic Wet Season (Howard Resh), zero nutrients (`cmbpfudbu00giq8z9jquzw746:zero_nutrients`) | 9/13 | Miss |
| Generic Wet Season (Howard Resh), calcium 20 mg l (`cmbpfudbu00giq8z9jquzw746:calcium_20_mg_l`) | 13/13 | Pass |
| Generic for Berries (Growing Edge), zero nutrients (`cmbpfucvl002kq8z9in7tpn5y:zero_nutrients`) | 12/12 | Pass |
| Generic for Berries (Growing Edge), calcium 20 mg l (`cmbpfucvl002kq8z9in7tpn5y:calcium_20_mg_l`) | 12/12 | Pass |
| Hoagland solution, zero nutrients (`cmbpfucun001uq8z9dgqeihsj:zero_nutrients`) | 12/12 | Pass |
| Hoagland solution, calcium 20 mg l (`cmbpfucun001uq8z9dgqeihsj:calcium_20_mg_l`) | 12/12 | Pass |
| Lettuce 2 (Howard Resh), zero nutrients (`cmbpfud8k00dhq8z9yz59ziwb:zero_nutrients`) | 13/13 | Pass |
| Lettuce 2 (Howard Resh), calcium 20 mg l (`cmbpfud8k00dhq8z9yz59ziwb:calcium_20_mg_l`) | 13/13 | Pass |
| Lettuce General (Howard Resh), zero nutrients (`cmbpfuddo00i2q8z9nldrxm9w:zero_nutrients`) | 13/13 | Pass |
| Lettuce General (Howard Resh), calcium 20 mg l (`cmbpfuddo00i2q8z9nldrxm9w:calcium_20_mg_l`) | 13/13 | Pass |
| Melons (Douglas Peckenpaugh), zero nutrients (`cmbpfud2n008cq8z99v63xpo5:zero_nutrients`) | 12/12 | Pass |
| Melons (Douglas Peckenpaugh), calcium 20 mg l (`cmbpfud2n008cq8z99v63xpo5:calcium_20_mg_l`) | 12/12 | Pass |
| Pepper (Howard Resh), zero nutrients (`cmbpfudb100fqq8z9wcljgmyi:zero_nutrients`) | 7/13 | Miss |
| Pepper (Howard Resh), calcium 20 mg l (`cmbpfudb100fqq8z9wcljgmyi:calcium_20_mg_l`) | 7/13 | Miss |
| Rice (Douglas Peckenpaugh), zero nutrients (`cmbpfud3e0092q8z9vq8gw25x:zero_nutrients`) | 11/13 | Miss |
| Rice (Douglas Peckenpaugh), calcium 20 mg l (`cmbpfud3e0092q8z9vq8gw25x:calcium_20_mg_l`) | 11/13 | Miss |
| Strawberry Drip Irrigation (schundler.com), zero nutrients (`cmbpfuctq0014q8z9li5oo75d:zero_nutrients`) | 7/12 | Miss |
| Strawberry Drip Irrigation (schundler.com), calcium 20 mg l (`cmbpfuctq0014q8z9li5oo75d:calcium_20_mg_l`) | 7/12 | Miss |
| Strawberry Fruiting (growing edge), zero nutrients (`cmbpfudel00iuq8z9l2wiwsfy:zero_nutrients`) | 12/12 | Pass |
| Strawberry Fruiting (growing edge), calcium 20 mg l (`cmbpfudel00iuq8z9l2wiwsfy:calcium_20_mg_l`) | 12/12 | Pass |
| Tomato (Howard Resh), zero nutrients (`cmbpfuda900f0q8z9hi9hu2gu:zero_nutrients`) | 11/12 | Miss |
| Tomato (Howard Resh), calcium 20 mg l (`cmbpfuda900f0q8z9hi9hu2gu:calcium_20_mg_l`) | 12/12 | Pass |
| Tomato Stage.1 - 10-14 days (Howard Resh), zero nutrients (`cmbpfud6400bbq8z9gwrj4cw7:zero_nutrients`) | 12/12 | Pass |
| Tomato Stage.1 - 10-14 days (Howard Resh), calcium 20 mg l (`cmbpfud6400bbq8z9gwrj4cw7:calcium_20_mg_l`) | 12/12 | Pass |
| Tomato Stage.2 - first cluster (Howard Resh), zero nutrients (`cmbpfud5900alq8z907fk2r92:zero_nutrients`) | 12/12 | Pass |
| Tomato Stage.2 - first cluster (Howard Resh), calcium 20 mg l (`cmbpfud5900alq8z907fk2r92:calcium_20_mg_l`) | 12/12 | Pass |
| Tomato Stage.3 - to plant maturity (Howard Resh), zero nutrients (`cmbpfud4a009vq8z953niqg47:zero_nutrients`) | 10/12 | Miss |
| Tomato Stage.3 - to plant maturity (Howard Resh), calcium 20 mg l (`cmbpfud4a009vq8z953niqg47:calcium_20_mg_l`) | 10/12 | Miss |
| Tomatoes - Fourth Cluster (U of Florida), zero nutrients (`cmbpfucxi0040q8z94o3m2bps:zero_nutrients`) | 12/12 | Pass |
| Tomatoes - Fourth Cluster (U of Florida), calcium 20 mg l (`cmbpfucxi0040q8z94o3m2bps:calcium_20_mg_l`) | 12/12 | Pass |
| Tomatoes - Second Cluster (U of Florida), zero nutrients (`cmbpfucz7005gq8z90ev9h6ao:zero_nutrients`) | 8/12 | Miss |
| Tomatoes - Second Cluster (U of Florida), calcium 20 mg l (`cmbpfucz7005gq8z90ev9h6ao:calcium_20_mg_l`) | 8/12 | Miss |
| Tomatoes - Third Cluster (U of Florida), zero nutrients (`cmbpfucyc004qq8z9mdajrg3w:zero_nutrients`) | 8/12 | Miss |
| Tomatoes - Third Cluster (U of Florida), calcium 20 mg l (`cmbpfucyc004qq8z9mdajrg3w:calcium_20_mg_l`) | 12/12 | Pass |
| Tomatoes - till First Cluster (U of Florida), zero nutrients (`cmbpfud030066q8z9fvdyru1m:zero_nutrients`) | 8/12 | Miss |
| Tomatoes - till First Cluster (U of Florida), calcium 20 mg l (`cmbpfud030066q8z9fvdyru1m:calcium_20_mg_l`) | 8/12 | Miss |
| Tomatoes - to termination (U of Florida), zero nutrients (`cmbpfucwh003aq8z9vmh9kner:zero_nutrients`) | 12/12 | Pass |
| Tomatoes - to termination (U of Florida), calcium 20 mg l (`cmbpfucwh003aq8z9vmh9kner:calcium_20_mg_l`) | 12/12 | Pass |
| Tropical Lettuce (Douglas Peckenpaugh), zero nutrients (`cmbpfud1t007mq8z9w16hx6hw:zero_nutrients`) | 12/12 | Pass |
| Tropical Lettuce (Douglas Peckenpaugh), calcium 20 mg l (`cmbpfud1t007mq8z9w16hx6hw:calcium_20_mg_l`) | 11/12 | Miss |
:::
Loading
Loading
0