Repository navigation
Add canonical module validation workflow - #135
Conversation
1023c20 to
c9f03f7
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Shared-runner changes can skip fixture validation, while missing commit/input provenance, unquoted artifact paths, and metadata type inference leave reproducibility gaps and cause checks to fail on valid inputs.
4 open issues
What changed in this PR
Adds opt-in validation for canonical OMIX modules, supporting #134 with reusable fixture execution and downloadable reviewer artifacts. OMIX-PCA-3D pilots the workflow without changing scientific behavior or publishing runtimes.
Changes:
- Introduces a validation manifest, generic planner/runner/report helper, and workflow using immutable runtime digests.
- Adds PCA fixture expectations and a local-only historical Plotly parity check.
- Documents validation commands, reviewer steps, runtime limitations, and ownership.
| File | Description |
|---|---|
tests/test_canonical_module_validation.py |
Tests planning, runner generation, reporting, and explicit discovery. |
templates/canonical-module-validation/validation.json |
Provides a module opt-in manifest template. |
scripts/canonical_module_validation.py |
Implements discovery, runtime-bound plans, runners, and artifact reports. |
modules/OMIX-PCA-3D/tests/validation.json |
Configures the PCA validation pilot and expected artifacts. |
modules/OMIX-PCA-3D/tests/test-pca-preparation.R |
Adds expected fixture coordinates and variance assertions. |
modules/OMIX-PCA-3D/tests/test-pca-historical-parity.R |
Adds explicit-path local comparison with historical Plotly output. |
modules/OMIX-PCA-3D/README.md |
Explains fixture validation and private-data handling. |
modules/OMIX-PCA-3D/CHANGELOG.md |
Records validation additions without scientific behavior changes. |
docs/README.md |
Links the validation guide. |
docs/canonical-module-validation.md |
Defines the validation contract, review steps, and responsibilities. |
.github/workflows/validate-canonical-module.yml |
Runs PR/manual validation and uploads review artifacts. |
Review effort: Balanced
| } | ||
|
|
||
| feature_matrix <- read_delimited(options$feature_matrix) | ||
| sample_metadata <- read_delimited(options$sample_metadata) |
There was a problem hiding this comment.
Addressed in d1560d3: the historical-parity reader now accepts column classes and reads sample metadata as character, matching the canonical CLI. The exact-container historical parity run passed afterward.
| "schema_version": 1, | ||
| "module": module, | ||
| "module_directory": f"modules/{module}", | ||
| "runtime_profile": profile, | ||
| "runtime_immutable_reference": reference, |
There was a problem hiding this comment.
Addressed in d1560d3: validation plans/reports now record the checked-out OMIX commit and SHA-256/byte identities for every declared committed fixture input. The end-to-end artifact report was regenerated successfully.
| "", | ||
| "# Fail clearly if the module did not produce every declared review artifact.", | ||
| ]) | ||
| lines.extend(f"test -e /artifacts/{artifact}" for artifact in plan["expected_artifacts"]) |
There was a problem hiding this comment.
Addressed in d1560d3: generated artifact existence checks now shell-quote the complete path via shlex.quote. A contract test covers an artifact name containing a space.
| check=True, | ||
| capture_output=True, | ||
| ) | ||
| requested = sorted({line.split("/")[1] for line in result.stdout.splitlines() if line.startswith("modules/") and len(line.split("/")) > 2}) |
There was a problem hiding this comment.
Addressed in d1560d3: changes to the shared helper, workflow, template, or helper test now select all opted-in modules. A contract test verifies the shared-helper case; module-only changes retain narrow discovery.
Copilot security finding addressedCopilot identified that GitHub expression interpolation into Bash could permit command substitution through
The refreshed validation fixture and all CodeQL/static-analysis checks pass. |

Closes #134
Scope
Evidence
python3 -m unittest tests/test_canonical_module_validation.pypassedghcr.io/nidap-community/omix-r-visualization@sha256:d0c00bf328a6a4897c243472db742860d20f7f700d7aab3bbe44dd96a7d5a842; it produced HTML, assets, coordinates, components, run summary, and checksum report3.836930773104541e-13at1e-10tolerance; point size 8 matchedReview boundaries
Human scientific-review scope
The human domain reviewer need review only the PCA scientific-evidence files and their artifacts—not the generic Python/Actions implementation. In Files changed, filter for
modules/OMIX-PCA-3D/and assess:tests/test-pca-preparation.R: the fixed public-fixture coordinates and variance are a valid regression oracle for the imported PCA sequence.tests/test-pca-historical-parity.R: it is test-only, preserves identifier formatting, uses explicit caller-supplied paths, and does not alter canonical PCA behavior.tests/validation.json: the public fixture invocation, selected samples, controls, and declared outputs are scientifically appropriate.README.md/CHANGELOG.md: validation and historical-parity claims are accurate and do not overstate runtime or platform equivalence.Validate canonical module / validate (OMIX-PCA-3D)artifact: inspect the coordinate/components tables, run summary, and HTML output.The generic validation helper, GitHub workflow, and security hardening are tooling scope. Copilot findings have been remediated and all static-analysis plus fixture checks pass. A scope-limited scientific-review comment is sufficient for the domain gate; use GitHub Approve only if the reviewer is also comfortable approving the entire PR.