Repository navigation
Bench: add JVM/WASI nightly lanes and qualify iOS/Android emulator coverage #858
Description
The nightly binding matrix currently covers Python, browser WASM, Node and C. Add Java/JVM and WASI, and assess Swift/iOS simulator and Android emulator lanes against physical-device performance before enabling them as representative mobile benchmarks.
This follows #839 (accepted runtime evidence and reusable harnesses), #840 (independent regression history and reporting), and #776 (coverage). It is a separate workflow expansion; the #840 runs validate the existing 427 native + 26 binding IDs.
Maintainer scope clarification (2026-11-05)
The physical-device/dedicated-bench restriction and macOS timing omission were specific to #839. They are not a blanket restriction on nightly benchmarks. A hosted iOS simulator or Android emulator may be included in nightlies if evidence shows sufficiently close performance to the corresponding physical-device workloads. This issue does not assume that condition is satisfied, and does not authorize treating emulator results as physical-device results without qualification.
Runtime scope
| Runtime | Starting point | Required work |
|---|---|---|
| Java/JVM | Merged JVM harness #21: 14 verified query/callback, preparation/registration, options, typed-log and storage cases | Add a Linux/JVM nightly job, adapt validated samples into the reporting contract, retain JDK/JVM flags and JIT warmup, and create separate history |
| WASI | Core WASI driver: empty REPL startup, startup + fixed script, persistent query round trip | Add a Wasmtime job with pinned runtime/configuration and actual WASI artifact provenance; preserve the documented startup/pipe overhead and in-memory contract |
| Swift/iOS | Merged Swift harness #16: 14 physical-device cases; currently rejects simulator timing | Evaluate an iOS simulator on the intended CI runner against physical iOS measurements before enabling a timing lane; retain physical mode and explicit platform identity |
| Android | Merged Android harness #17: 14 physical-device cases | Evaluate a pinned emulator/ABI/system image on the intended CI runner against physical Android measurements before enabling a timing lane |
Browser WASM does not measure WASI, JVM does not measure Android ART, and shared UniFFI code does not measure each foreign runtime.
Mobile qualification gate
Apple says simulators do not replicate physical-device performance, and Android discourages emulator benchmarking because measurements depend on the host OS/hardware. See Apple's device/simulator guidance and Android benchmarking in CI. These are reasons to require evidence, not a claim that Minigraf's workloads have already failed qualification.
Before measuring, define and obtain review of what “sufficiently close” means for each workload category. Use matched core/binding/harness revisions, Release builds, equivalent cases and timing boundaries, repeated independent runs, and recorded device/runner/OS/ABI/toolchain/JIT/thermal conditions. Assess absolute latency differences and run-to-run variation, plus whether representative real performance changes produce comparable regression signals. Correlation or a constant scaling factor alone does not establish close physical-device timings. Validate exact results on every invocation.
Qualify compute/callback, allocation/conversion and storage/startup cases separately. Enable only the cases and configurations supported by the evidence. If qualification fails, retain the explicit exclusion and propose a physical-device lane; a simulator-only trend monitor requires a separately approved, clearly labelled purpose and must not silently satisfy the physical-fidelity condition.
Acceptance criteria
- Add Java/JVM and WASI to the default-branch-owned nightly workflow with separate main/v3 companion PRs for shared CI changes. Resolve the measured core once, pin binding/driver/runtime revisions, and use the actual measured core rather than a stale binding release pin.
- Reuse the validated harnesses; review and document the exact nonempty shortlist and timing boundaries. Adapt their raw samples into the normal selected/executed/uploaded reporting contract without fabricated or replayed observations.
- Publish reproducible physical-versus-simulator/emulator qualification for iOS and Android, with reviewed numerical closeness/repeatability criteria and per-category decisions. Add qualifying lanes; explicitly disposition any rejected cases/platforms.
- Give every runtime/platform/ABI/runner configuration its own appropriately versioned testbed/history. Keep simulator/emulator and physical-device data separate. Preserve main/v3 history across main advances and do not count inherited observations as independent runs.
- Establish usable numeric comparisons from fresh independent runs for every added nightly ID; verify the shared t-test upper probability 0.999 / maximum sample size 30 without interpreting either as a latency percentage budget or 30 completed runs.
- Verify exact selected = executed = uploaded sets, source hashes, failure handling, branch/suite-scoped notification integration and actual scheduled routing. Retain raw samples, reports, models, runtime/runner metadata and API evidence beyond Actions artifact expiry.
- Document duration, concurrency, runner availability/cost and repeatability limits; run heavy local work on the idle dedicated bench, not the laptop. Synchronize benchmark coverage/status docs and clean task scratch after archival.
No database runtime/API/file-format change is needed. This is reliability and cross-platform performance verification under PHILOSOPHY.md.
Activity