8000 Add canonical module validation workflow by maggiecam · Pull Request #135 · NIDAP-Community/OMIX · GitHub
[go: up one dir, main page]

Skip to content

Add canonical module validation workflow - #135

Merged
maggiecam merged 3 commits into
mainfrom
beacon/canonical-module-validation
Oct 29, 2026
Merged

maggiecam merged 3 commits into
mainfrom
beacon/canonical-module-validation

Conversation

@maggiecam
@maggiecam maggiecam commented Oct 28, 2026 •
Copy link
Copy Markdown
Member

Closes #134

Scope

  • add an opt-in module validation manifest, generic planner/runner/report tool, and PR workflow that runs against immutable runtime digests
  • upload reviewable fixture artifacts plus checksums; do not build, publish, tag, or release any runtime
  • pilot the contract with OMIX-PCA-3D, including exact public-fixture results and an explicit-path local historical Plotly parity test
  • document contributor/reviewer steps and role ownership

Evidence

  • python3 -m unittest tests/test_canonical_module_validation.py passed
  • generic PCA fixture runner passed in ghcr.io/nidap-community/omix-r-visualization@sha256:d0c00bf328a6a4897c243472db742860d20f7f700d7aab3bbe44dd96a7d5a842; it produced HTML, assets, coordinates, components, run summary, and checksum report
  • explicit-path local historical comparison in that same image: 9 samples / 27 coordinates, maximum absolute difference 3.836930773104541e-13 at 1e-10 tolerance; point size 8 matched
  • no private input or output is committed or uploaded

Review boundaries

  • the base r-visualization lane does not include the module-local edgeR overlay, so its CPM assertion is intentionally skipped and this PR makes no effective-overlay-equivalence claim
  • the initial GitHub run tests the workflow path and artifact handoff; it does not claim Code Ocean, effective-overlay, or cross-environment equivalence

Human scientific-review scope

The human domain reviewer need review only the PCA scientific-evidence files and their artifacts—not the generic Python/Actions implementation. In Files changed, filter for modules/OMIX-PCA-3D/ and assess:

  1. tests/test-pca-preparation.R: the fixed public-fixture coordinates and variance are a valid regression oracle for the imported PCA sequence.
  2. tests/test-pca-historical-parity.R: it is test-only, preserves identifier formatting, uses explicit caller-supplied paths, and does not alter canonical PCA behavior.
  3. tests/validation.json: the public fixture invocation, selected samples, controls, and declared outputs are scientifically appropriate.
  4. README.md / CHANGELOG.md: validation and historical-parity claims are accurate and do not overstate runtime or platform equivalence.
  5. The successful Validate canonical module / validate (OMIX-PCA-3D) artifact: inspect the coordinate/components tables, run summary, and HTML output.

The generic validation helper, GitHub workflow, and security hardening are tooling scope. Copilot findings have been remediated and all static-analysis plus fixture checks pass. A scope-limited scientific-review comment is sufficient for the domain gate; use GitHub Approve only if the reviewer is also comfortable approving the entire PR.

@maggiecam
maggiecam force-pushed the beacon/canonical-module-validation branch from 1023c20 to c9f03f7 Compare October 28, 2026 20:41
@maggiecam
maggiecam requested a balanced review from Copilot October 28, 2026 20:44
Copilot AI left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Shared-runner changes can skip fixture validation, while missing commit/input provenance, unquoted artifact paths, and metadata type inference leave reproducibility gaps and cause checks to fail on valid inputs.

4 open issues
What changed in this PR

Adds opt-in validation for canonical OMIX modules, supporting #134 with reusable fixture execution and downloadable reviewer artifacts. OMIX-PCA-3D pilots the workflow without changing scientific behavior or publishing runtimes.

Changes:

  • Introduces a validation manifest, generic planner/runner/report helper, and workflow using immutable runtime digests.
  • Adds PCA fixture expectations and a local-only historical Plotly parity check.
  • Documents validation commands, reviewer steps, runtime limitations, and ownership.
File Description
tests/​test_canonical_module_validation.py Tests planning, runner generation, reporting, and explicit discovery.
templates/​canonical-module-validation/​validation.json Provides a module opt-in manifest template.
scripts/​canonical_module_validation.py Implements discovery, runtime-bound plans, runners, and artifact reports.
modules/​OMIX-PCA-3D/​tests/​validation.json Configures the PCA validation pilot and expected artifacts.
modules/​OMIX-PCA-3D/​tests/​test-pca-preparation.R Adds expected fixture coordinates and variance assertions.
modules/​OMIX-PCA-3D/​tests/​test-pca-historical-parity.R Adds explicit-path local comparison with historical Plotly output.
modules/​OMIX-PCA-3D/​README.md Explains fixture validation and private-data handling.
modules/​OMIX-PCA-3D/​CHANGELOG.md Records validation additions without scientific behavior changes.
docs/​README.md Links the validation guide.
docs/​canonical-module-validation.md Defines the validation contract, review steps, and responsibilities.
.github/​workflows/​validate-canonical-module.yml Runs PR/manual validation and uploads review artifacts.

Review effort: Balanced

}

feature_matrix <- read_delimited(options$feature_matrix)
sample_metadata <- read_delimited(options$sample_metadata)
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in d1560d3: the historical-parity reader now accepts column classes and reads sample metadata as character, matching the canonical CLI. The exact-container historical parity run passed afterward.

Comment on lines +110 to +114
"schema_version": 1,
"module": module,
"module_directory": f"modules/{module}",
"runtime_profile": profile,
"runtime_immutable_reference": reference,
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in d1560d3: validation plans/reports now record the checked-out OMIX commit and SHA-256/byte identities for every declared committed fixture input. The end-to-end artifact report was regenerated successfully.

Comment thread scripts/canonical_module_validation.py Outdated
"",
"# Fail clearly if the module did not produce every declared review artifact.",
])
lines.extend(f"test -e /artifacts/{artifact}" for artifact in plan["expected_artifacts"])
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in d1560d3: generated artifact existence checks now shell-quote the complete path via shlex.quote. A contract test covers an artifact name containing a space.

Comment thread scripts/canonical_module_validation.py Outdated
check=True,
capture_output=True,
)
requested = sorted({line.split("/")[1] for line in result.stdout.splitlines() if line.startswith("modules/") and len(line.split("/")) > 2})
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in d1560d3: changes to the shared helper, workflow, template, or helper test now select all opted-in modules. A contract test verifies the shared-helper case; module-only changes retain narrow discovery.

@maggiecam
Copy link
Copy Markdown
Member Author

Copilot security finding addressed

Copilot identified that GitHub expression interpolation into Bash could permit command substitution through inputs.module or an unvalidated matrix module name. Commit b99bff9 resolves this by:

  • validating module names against the canonical [A-Za-z0-9][A-Za-z0-9_-]* contract before discovery emits a matrix entry;
  • rejecting invalid manual module input;
  • passing dispatch, PR, matrix, image, and artifact values into shell steps through environment variables and quoting their shell expansions; and
  • adding a regression test with a shell-syntax-shaped changed module name.

The refreshed validation fixture and all CodeQL/static-analysis checks pass.

@maggiecam
maggiecam merged commit 6d01613 into main Oct 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[QA]: add reusable canonical-module PR validation and reviewer artifacts

2 participants

0