Repository navigation
Pywr-DRB v2.2.0 - #102
Merged
Merged
Pywr-DRB v2.2.0#102
Conversation
Collaborator
- Add STARFITOfflineSimulator for offline STARFIT release pre-simulation
- Add perfect_foresight prediction mode to preprocessing pipeline
- Add flow_prediction_mode option to ModelBuilder
- Regenerate prediction data with perfect_foresight columns
- Add flow_prediction_mode to Options docstring
- Add clarifying comments to predict_timeseries._predict_value
- Fix STARFITOfflineSimulator to include catchment consumption in storage balance
- USe global Id in starfit.py
- Restore comm arg in preprocessors to enable MPI
- Attempt to fix str coercsion in ensemble preprocessor
- Attempt to fix str coercsion in ensemble preprocessor
- Attempt MPI fix part N
- Add main block to obs_data_retrieval.py for direct execution
- Fix ensemble index in lower basin ffmp
- Fixing obs storage data and pulling observations to May 2026
- Fix test obs loader
- Add custom starfit tuning ability and experiments/ folder
- Update tuning notebook
- Prepping for v2.2beta: Update docs with better API coverage, fix some minor argument docstrings, update pyproject.toml
- Updates for 2.2: update predicted_inflow_mgd.csv files to contain perfect foresight data | remove gage_flow as prediction_mode throughout | update release notes
- New STARFIT default params for blueMarsh, beltzville, fewalter and prompton | updated predicted_inflows_mgd.csv using the new STARFIT operations to inform prediction
- Add experiments/v2.2beta_testing
- Update beta testing
- Prepare v2.2.0 release: STARFIT comparison figure, docs, packaging, test fixes
- Untrack experiments/ folder for v2.2 release
Adds a new module that pre-simulates STARFIT reservoir releases offline (without running the full Pywr model). This is a prerequisite for the perfect_foresight flow prediction mode, which uses pre-simulated releases to create more accurate retrospective inflow predictions at Montague and Trenton. Also adds the independent_starfit_reservoirs list to utils/lists.py and updates pre/__init__.py exports.
Updates the prediction pipeline to support the new perfect_foresight mode: predict_timeseries.py (base class): - Switch from positional to date-based indexing (fixes bug when data is subset) - Add gage_flow and perfect_foresight mode handling - Skip regression training when no regression modes are used - Add data validation in _fit_regression() predict_inflows.py: - Add STARFIT-aware perfect_foresight mode that uses pre-simulated releases - For perfect_foresight: NYC reservoirs return 0, STARFIT reservoirs use simulated releases, non-reservoirs use catchment inflows with WC adjustment - Add gage_flow mode for direct gage data lookup - Update ensemble preprocessor with STARFIT support predict_diversions.py: - Narrow supported modes to regression_disagg and perfect_foresight - Add comm parameter to ensemble preprocessor for MPI flexibility
Moves flow_prediction_mode from a module-level variable to the Options
dataclass, enabling per-instance configuration. Default remains
"regression_disagg" for backward compatibility.
Users can opt into perfect_foresight mode via:
ModelBuilder(..., options={"flow_prediction_mode": "perfect_foresight"})
Adds validation in make_model() to catch invalid mode values.
Adds run_inflow_prediction.py script and regenerates all predicted inflow and diversion CSV files with both regression_disagg and perfect_foresight columns for all 9 flow types. prediction data now includes columns like: delMontague_lag1_perfect_foresight delTrenton_lag2_perfect_foresight demand_nj_lag1_perfect_foresight regression_disagg predictions also updated due to date-based indexing fix in the preprocessing pipeline.
Documents the relationship between the base class perfect_foresight behavior and the PredictedInflowPreprocessor override, and notes the date-based indexing strategy that prevents subsetting bugs.
…ge balance The offline simulator now loads catchment water consumption data and uses net inflow (gross - consumption) for the storage balance calculation, matching the Pywr model's mass balance where catchment flow is split between reservoir inflow and consumption nodes. The release formula still uses gross inflow, matching the online STARFITReservoirRelease. Also removes the legacy generate_presimulated_releases() function (HDF5 extraction) which is not needed for this branch. All prediction data regenerated with corrected STARFIT releases.
Adds the same __main__ entry point present on nyc_opt so the module can be run directly via `python -m pywrdrb.pre.obs_data_retrieval`.
… minor argument docstrings, update pyproject.toml
…fect foresight data | remove gage_flow as prediction_mode throughout | update release notes
…ompton | updated predicted_inflows_mgd.csv using the new STARFIT operations to inform prediction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.