Repository navigation
bench: record JVM, Android, Swift/iOS and WASI release evidence - #852
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
5 tasks done
adityamukho
marked this pull request as ready for review
November 5, 2026 02:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the missing #839 runtime evidence: three timing series each for JVM/WASI on the dedicated Linux bench and Android/Swift-iOS on physical AWS Device Farm devices. Retains all 1,350 samples across 45 cases, 135 case/run summaries, exact build provenance, lockfiles, checksums and available conditions. Every timed invocation checks actual results; this is source-build evidence, not published-package certification.
New #839 timings use only Device Farm/the bench. GitHub builds and checks correctness; macOS timing is excluded. The existing 26 Python/browser/Node/C nightly metrics and workflows are unchanged. The remaining API/lifecycle dispositions explicitly identify unmeasured costs and proposed exclusions awaiting acceptance.
Validation: all twelve timing datasets and 34 artifact checksums verified; all 135 summaries recomputed. Seven oracle/optimization-guard tests pass. Earlier JVM/Android/WASI validation rejected 36 malformed variants; iOS validation rejected twelve additional malformed variants. Swift simulator and physical smoke/oracle tests passed, and all three physical timing series contain fourteen cases and 140 samples. Packaged artifact identities match the retained build records. Companion binding CI and consumer tests pass.
Limitations remain explicit: public Device Farm selects model/OS, not a reserved physical unit; idle/power conditions are not verified and thermal observations are snapshots. Some results vary substantially. The first iOS smoke job also ran a valid benchmark after smoke; later runs used the corrected method filter. iOS customer-file attachment failed, so complete JSONL streams were recovered from retained automation logs. No outliers were removed, no latency/RSS threshold is claimed, and private bench/account details are omitted. Bench scratch was cleaned after verified archival; original evidence is retained locally.
Review results and collection details and proposed dispositions. Companion drafts: JVM #21, Android #17, Swift/iOS #16.
The requesting maintainer will provide both binding and release sign-offs after reviewing the completed evidence. Acceptance of variability, device-selection limits and each proposed disposition remains pending. Relates to #839; does not close it.