8000 Irrigation EC M2a: bounded estimates for species-only coverage gaps by rafatahmed · Pull Request #29 · rafatahmed/Flahax · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
## Unreleased

- Add experimental irrigation EC25 from speciation (McCleskey 2012 Eq. 8) with explicit partial results, envelope checks and provenance; no accuracy claim.
- Irrigation EC (M2a): species without a McCleskey coefficient now give a bounded `estimated` result (`bounded`, `ec25_range_ms_cm`, midpoint `ec25_ms_cm`, `uncertainty_ms_cm`, audit `uncovered_bound_ms_cm`; upper limit from 80 S cm2/mol per charge) instead of `partial`; `MODEL_VERSION` is now m2a and `compare_measured_ec` reports `ec25_range_ms_cm` and `within_range`.
- Add the material traceability API (`docs/materials.md`): shipped identity records (`load_identities`, `identity_for_product`), user grades and lots (`make_grade`, `make_lot`), `resolve_material` (per-nutrient basis, band ends, flags, provenance) and `nutrient_band`; `catalogue_dose_totals` and `calculate_irrigation_ec` take `materials=`, undisclosed blends raise or report `composition_undisclosed` in chemistry, and EC and equilibrium-delivery audits carry material provenance.
- Link every catalogue product in `library.json` to its chemical identity (`identityId`) and record its `analysisBasis` (`declared_catalogue`, `identity_theoretical` or `grade_typical`) and `source` document reference. Product `id` and `name` values are unchanged.
- Correct Mg Nitrate to Mg(NO3)2·6H2O theory (owner decision D1): formula `Mg(NO3)2` → `Mg(NO3)2.6H2O`; N_NO3 11 → 10.92564 %, Mg 9.65 → 9.47931 %; basis `identity_theoretical`. Its declared-form charge residual at 1 g/kg water goes from +0.5532 % to ~0 %, so it no longer trips the EC O5 gate on its own.
Expand Down
60 changes: 53 additions & 7 deletions docs/conductivity.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,10 @@ still be reported when EC evidence is missing; that does not approve the recipe
or authorize EC-driven correction. Verify EC after preparation/dilution using
meters rated for the relevant stock and irrigation ranges.

## Calculated irrigation EC (experimental, M1)
## Calculated irrigation EC (experimental, M2a)

Purpose: estimate the EC of an irrigation solution before mixing, and compare
it with a meter reading after mixing. This is **experimental (M1)** and Python
it with a meter reading after mixing. This is **experimental (M2a)** and Python
only: there is no CLI flag and no grower-facing number. It is separate from the
calibration-based planner above.

Expand All @@ -141,8 +141,8 @@ applied.

| Status | Meaning |
|---|---|
| `estimated` | Every charged species above 1e-12 mol/kgw has a McCleskey coefficient. `ec25_ms_cm` is the estimate. |
| `partial` | Some charged species have no coefficient (listed in `missing_species`), or, with a `supplied` pH, there are unrecorded counterions (`unrecorded_counterions`, rule O5) reported in eq/kgw. Only a known-species subtotal is reported, with no total. |
| `estimated` | Every charged species above 1e-12 mol/kgw has a McCleskey coefficient, or the species without one are covered by the bound below. `ec25_ms_cm` is the estimate. |
| `partial` | With a `supplied` pH, there are unrecorded counterions (`unrecorded_counterions`, rule O5) reported in eq/kgw. Only a known-species subtotal is reported, with no total. |
| `not_supported` | Outside the envelope (temperature, ionic strength, non-closed carbon) or the assay charge residual is too large. `reason` says which; no EC is reported. |

Assay charge residual (rule O5): a residual up to 0.1 % of the absolute
Expand All @@ -153,9 +153,55 @@ it is over the speciated absolute equivalents. Above it, a `supplied` pH gives
`partial` (`unrecorded_counterions`) and a `calculated` pH gives
`not_supported` (`assay_charge_residual_too_large`).

Phosphate recipes are `partial` in M1: H2PO4-, and also MgOH+, CaOH+ and
charged chelates, have no McCleskey coefficient, so most phosphate, Mg and Ca
solutions return `partial` until M2.
Bounded estimates (decision O2): charged species with no McCleskey coefficient
(H2PO4-, MgOH+, CaOH+, charged chelates) no longer make the result `partial`.
Their contribution is non-negative, so the known subtotal is a lower limit. The
upper limit adds sum(|z| x 80 x m) over those species, because no ion's limiting
molar conductivity exceeds 80 S cm2/mol per unit charge at infinite dilution
(Adamson 1973 p. 512; the largest non-H+/OH- ion is SO4-2 at 79.8). The result is
`estimated` with `bounded = True`:

- `ec25_range_ms_cm`: `(known subtotal, known subtotal + bound)`; the range extends only upward from the known subtotal, because missing contributions cannot be negative.
- `ec25_ms_cm`: the midpoint; `uncertainty_ms_cm`: half the bound.
- `audit["uncovered_bound_ms_cm"]`: the bound; `missing_species` still lists every uncovered species.
- `transport_numbers` is `{}`, and `MODEL_VERSION` is `irrigation-ec-mccleskey2012-m2a-experimental`.

The midpoint and `uncertainty_ms_cm` describe only the uncovered-species interval; they are not an accuracy statement and exclude model error. Because 80 S·cm²/mol per charge is conservative, the true value tends to lie in the lower half of the range.

`partial` arises only from unrecorded counterions; temperature, carbon,
ionic-strength domain and undisclosed blends give `not_supported`. All 26 shipped formulas return `estimated`, with ranges up to 14.4 % of
the midpoint (Generic Wet Season). Phosphate coefficients are not yet
measured, so phosphate stays under the bound until M2b. `compare_measured_ec`
also returns `ec25_range_ms_cm` and `within_range` (True, False, or None when the
prediction is not bounded); it is a fact, not a pass/fail verdict.

Example (KH2PO4 at 1 g/kg water, calculated pH):

```python
from flahax import calculate_ec_from_totals, compare_measured_ec

m = 1 / 136.09 # 1 g/kg water of KH2PO4, mol/kgw
result = calculate_ec_from_totals(
{"K+": m, "PO4-3": m}, solution_id="kh2po4-1g",
ph_mode="calculated", phase_policy="dissolved")
print(result.status, result.bounded)
print([round(x, 3) for x in result.ec25_range_ms_cm])
print(round(result.ec25_ms_cm, 3), round(result.uncertainty_ms_cm, 3))
print([s["name"] for s in result.missing_species])
check = compare_measured_ec(result, measured_ec25_ms_cm=0.80,
meter_reference_temperature_c=25)
print(check["within_range"])
```

Output:

```text
estimated True
[0.524, 1.113]
0.818 0.295
['H2PO4-', 'HPO4-2', 'KHPO4-', 'PO4-3']
True
```

```python
from flahax import calculate_ec_from_totals
Expand Down
2 changes: 1 addition & 1 deletion docs/development-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The matrix below describes the 0.3.1 capabilities and their boundaries.
| Industrial acid selection and product mass/volume | Standard workflow HNO3/H3PO4, `acid_selection.py`, four preserved lower-level PHREEQC acid fixtures | Supplied assay/density/provenance, nutrient and volume limits; no automatic dosing approval |
| Source-specific CLI | `planning_cli.py`, `__main__.py`, `test_site_planning.py` | Explicit JSON/interactive inputs, no default water or equipment model |
| No-new-measurements EC screening | `calculated_ec.py`, exact SQM anchors with source hash, regression tests | Pre-acid dilute approximation at 25 C, restricted loading, unknown accuracy; incomplete products yield no total |
| Calculated irrigation EC (McCleskey 2012 on speciation) | `irrigation_ec.py`, transcription and rule tests | Experimental M1; phosphate and chelate species partial; no measured accuracy |
| Calculated irrigation EC (McCleskey 2012 on speciation) | `irrigation_ec.py`, transcription and rule tests | Experimental M2a; phosphate and chelate species give a bounded range; no measured accuracy |
| Optional measured-profile EC method | `conductivity.py`, synthetic regression profiles | Requires matching measured evidence when used; example profiles are fictional |
| A/B and acid report clarity | `pepper_stock_review.py`, HTML/Markdown/JSON examples and tests | Candidate grouping only; trace locations unresolved; no approved full pepper StockPlan |
| Stock planning API | Existing sourced compatibility/solubility/capacity checks | Caller must supply complete applicable records; absence from a pairwise table is not evidence of compatibility |
Expand Down
101 changes: 101 additions & 0 deletions docs/superpowers/plans/2026-10-19-irrigation-ec-m2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Irrigation EC M2 Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make shipped recipes return `estimated` irrigation EC. M2a publishes an O2 bound on uncovered species. M2b adds phosphate coefficients fitted to measured data (O1).

**Architecture:**
- **M2a:** changes `irrigation_ec._ec_from_state`. Uncovered charged species no longer force `partial`. They add a one-sided bound and the result carries an EC range.
- **M2b:** adds a phosphate section to the conductivity data file. The coefficients are produced by a fitting tool from owner-supplied measured data, so phosphate species become covered and the range narrows.
- **Shared:** both stages reuse the existing statuses, audit record and post-mix comparison.

**Tech Stack:** Python 3.13, standard library only (the fit uses a small Gauss–Newton or grid search written in `tools/`, not SciPy), `unittest`.

**Spec:** [docs/superpowers/specs/2026-10-18-irrigation-ec-mccleskey-design.md](../specs/2026-10-18-irrigation-ec-mccleskey-design.md), §11 and §13.

## Global Constraints

- **λ_max:** 80 S·cm²/mol per unit charge, at infinite dilution, with no ionic-strength reduction. The constant `LAMBDA_MAX_PER_CHARGE = 80.0` carries its citation: limiting-conductivity table, Adamson 1973 p. 512; max non-H⁺/OH⁻ is SO₄²⁻ 8000 79.8.
- **Bound:** Σ |zᵢ|·80·mᵢ over uncovered charged species above 1e-12 mol/kgw, in mS/cm (the same units identity as Eq. 8). Every uncovered species is still listed.
- **Bounded estimate:** `status='estimated'`, `bounded=True`, `ec25_range_ms_cm=(known, known+bound)`, `ec25_ms_cm=known+bound/2`, `uncertainty_ms_cm=bound/2`, audit `uncovered_bound_ms_cm`.
- **Rules O2 does not override:** unrecorded counterions, the O5 assay gate, temperature, carbon and the ionic-strength domain keep their existing statuses.
- **Post-mix comparison:** reports `within_range` as a fact, with no pass/fail and no tolerance.
- **No invented parameters:** phosphate coefficients exist only after M2b runs on owner-supplied data. Until then phosphate stays under the bound.
- **Qualifier unchanged:** "Accuracy on this fertilizer mixture has not yet been established."
- **Branch:** `feat/ec-m2`, stacked on `docs/m1-signoff` (rafatahmed/Flahax#28). Stage only named files. Never `git stash`. Never stage README.md, docs/publication/*, docs/landscape-comparison-swot.md or outputs/.
- **Python path:** use `PYTHONPATH="src;."`, never `src` alone, which picks up an installed old package.

## Review Focus

1. **A recipe whose only problem is uncovered species** gives `estimated` with a range, never `partial`. A recipe with unrecorded counterions *and* uncovered species stays `partial` (unrecorded_counterions). Pinned in Task 1.
2. **No uncovered species.** `bounded=False`, `ec25_range_ms_cm=None` and the EC is identical to M1 (KCl 0.01 still 1.42349). Pinned in Task 1.
3. **Post-mix comparison on a bounded estimate.** The difference is to the midpoint, `within_range` is reported, and partial is still refused. Pinned in Task 2.
4. **Band ends:** each end carries its own bound and range. Pinned in Task 2.

---

### Task 1 (M2a): O2 bound in the EC result

**Files:** Modify `src/flahax/irrigation_ec.py`. Test: `tests/test_irrigation_ec.py` (class `BoundTests`, plus existing tests updated where `partial` / `missing_species_contributions` becomes bounded `estimated`).

**Interfaces:**
- `LAMBDA_MAX_PER_CHARGE: float = 80.0`.
- `IrrigationECResult` gains `bounded: bool = False`, `ec25_range_ms_cm: tuple | None = None`, `uncertainty_ms_cm: float | None = None` and `uncovered_bound_ms_cm: float | None = None`.
- The audit gains `uncovered_bound_ms_cm` and `lambda_max_per_charge`.

- [ ] **Step 1: Failing tests.**
- `test_kh2po4_is_bounded_estimate`: totals K+ = PO4-3 = 1/136.09 mol/kgw, calculated mode.
- Status `estimated` and `bounded`.
- Range lower equals the M1 known subtotal (≈0.524, delta 0.002).
- Upper = lower + Σ|z|·80·m over the reported missing species, recomputed in the test from `missing_species`.
- Midpoint and uncertainty are consistent.
- `test_unbounded_result_unchanged` (Review Focus 2): KCl 0.01 gives `bounded=False`, `ec25_range_ms_cm=None` and EC 1.42349 (places 4).
- `test_unrecorded_counterions_still_partial` (Review Focus 1): KH₂PO₄ totals at supplied pH 4.0 stay `partial` / `unrecorded_counterions`.
- `test_all_shipped_formulas_estimated`: all 26 shipped formulas (recommend doses, g/L as g/kgw, DeclaredPureWater, calculated mode) return `estimated`, and each range width / midpoint is ≤ 0.12. The audit's worst case is about 5.5 % for others plus about 8 % for phosphate. The test asserts the measured maximum with margin; record the actual values in the report.
- [ ] **Step 2:** Update the existing tests that asserted `partial` / `missing_species_contributions` on species-only cases (for example the Mg/SO₄ pair test and the phosphate test). Assert the bounded estimate and keep the missing-species assertions. Mark each change with a comment `# M2a O2 bound`.
- [ ] **Step 3:** Implement. Run `tests.test_irrigation_ec`, then the full suite.
- [ ] **Step 4:** Commit with the message "Publish an O2 bound so species-only gaps give a bounded EC estimate".

### Task 2 (M2a): Comparison and band ends with bounded estimates

**Files:** Modify `src/flahax/irrigation_ec.py` (`compare_measured_ec`, `calculate_irrigation_ec_band`). Test: `tests/test_irrigation_ec.py`.

**Rules:**
- `compare_measured_ec` adds the keys `ec25_range_ms_cm` and `within_range` (bool or None for unbounded), and keeps the difference to `ec25_ms_cm`.
- Band ends need no logic change; add tests that each end carries its own bound.

- [ ] **Failing tests:**
- `test_compare_bounded`: inside and outside the range.
- `test_compare_unbounded_has_none_within_range`.
- `test_band_ends_carry_own_bounds` (Review Focus 4): a graded KH₂PO₄-like case where the ends differ.
- [ ] **Implement, run, full suite, then commit** with the message "Report range membership for bounded EC estimates".

### Task 3 (M2b): Phosphate coefficients from measured data. BLOCKED until the owner supplies data.

**Precondition (controller checks; if missing, stop and report BLOCKED):**
- `tests/fixtures/conductivity/crc_phosphate_lambda0.json`: owner-checked CRC λ° for H2PO4-, HPO4-2 and PO4-3, with the edition and page.
- `tests/fixtures/conductivity/muccitelli1994_25c.json` and an independently typed copy `..._independent.json`. They hold every 25 °C KH₂PO₄ and K₂HPO₄ point (concentration in mol/L, κ in µS/cm or as printed, with units) from Muccitelli & DiAngelo (1994), plus the paper's DOI and the PDF SHA-256.

**Files:**
- Create: `tools/fit_phosphate_conductivity.py` (reads the fixtures, converts mol/L to mol/kgw with a stated density of water at 25 °C, speciates with FlahaX, and fits A and B per phosphate ion by least squares on κ with λ° fixed).
- Modify: `src/flahax/data/conductivity_mccleskey2012.json`. Add a top-level `"supplementary"` section with `{ion: {lambda0_25c, A_25c, B, source, fit_rms_percent, points}}`. Do not add these ions to the McCleskey `ions` table, because they are not in the paper.
- Modify: `irrigation_ec.ionic_molal_conductivity` / `covered_species`, so supplementary ions are covered at 25 °C only.
- Test: `tests/test_irrigation_ec.py` (class `PhosphateCoefficientTests`).

- [ ] **Failing tests:**
- `test_fixture_transcriptions_agree`;
- `test_fit_reproduces_measurements` (each point within the published RMS, which is ≤ 2 %);
- `test_phreeqc_cross_check` (CRC λ° within 15 % of the `phreeqc.dat`-derived 31.8 / 103.6 / 206.8; report, don't hide);
- `test_kh2po4_now_covered`: KH₂PO₄ totals give an unbounded `estimated` result, or a much smaller bound if minor pairs remain.
- [ ] **Run the tool**, commit its output, then run the full suite. Recount the 26 formulas and record the narrowed ranges in the design §13.
- [ ] **Commit** with the message "Add phosphate conductivity coefficients fitted to measured data".

### Task 4: Docs, exports, verification

**Files:**
- `docs/conductivity.md`: the bounded estimate, its meaning and the range fields. Include one runnable example, pasting the real output.
- `CHANGELOG.md`: one `## Unreleased` line per stage.
- `src/flahax/data/API_REFERENCE.md`: regenerate.

- [ ] **Full checks:** API `--check`, `tests.test_release_hygiene`, the full suite (foreground, timeout 600000 ms, never two concurrent runs), `compileall` and `git diff --check`.
- [ ] **Commit** with the message "Document bounded irrigation EC estimates".
Loading
Loading
0