adityamukho · Nov 5, 2026 · Nov 4, 2026 · Nov 4, 2026 · Nov 4, 2026 · Nov 4, 2026
@@ -0,0 +1,2 @@
+/build/
+/__pycache__/
@@ -0,0 +1,93 @@
+# Manual runtime evidence (#839)
+
+The [scope and coverage plan](../../docs/benchmarks/839-bindings/README.md) and
+[initial results](../../docs/benchmarks/839-bindings/results.md) distinguish these
+manual runs from existing GitHub nightly benchmarks. No release acceptance or
+latency threshold is implied.
+
+## WASI reproduction
+
+Use a clean core checkout at the recorded revision and Rust 1.99.1. Copy the
+retained `wasi-Cargo.lock` into that disposable checkout as `Cargo.lock`, then:
+
+```sh
+rustup target add wasm32-wasip1
+cargo build --locked --release --target wasm32-wasip1 --bin minigraf
+python3 -m unittest discover -s benchmarks/bindings839 -p 'test_*.py'
+python3 benchmarks/bindings839/wasi.py --runtime /path/to/wasmtime-49.0.2 \
+  --wasm target/wasm32-wasip1/release/minigraf.wasm \
+  --core-revision 33e5dffadebe0124e6287fc91cede4ff2b7a2b2f \
+  --output /fresh/output/smoke --mode smoke
+```
+
+The script requires a UTF-8 process locale, as used in the recorded runs, and
+rejects Python `-O` explicitly so assertions cannot silently disappear. Run
+the `benchmark` mode only on the dedicated bench, serially with CPU and memory
+affinity on one NUMA node and no concurrent builds. Use three fresh output
+directories. Record idle checks and toolchain/artifact hashes externally as well
+as the emitted environment record: `--core-revision` is an asserted label, not
+proof of a source build. The retained series additionally verifies all core
+compilation inputs and the actual WASM/driver against their recorded hashes.
+
+Startup includes uncached compilation. A persistent query includes pipe and
+line-framing overhead. Results must have exact rows, count and sum, with successful
+exit and empty stderr. This is the actual REPL contract, not a prepared/callback
+binding API. Run `validate.py wasi OUTPUT/samples.jsonl` after each timed series.
+
+## JVM and Android
+
+Their binding repositories contain the source-build scripts and harnesses under
+`bench/manual` and `bench/device-farm`. Run the JVM installed application in three
+fresh processes with `-Xms512m -Xmx512m -XX:+UseG1GC` (the generated launcher sets
+these), on the same idle NUMA-pinned bench. The read-only-open case deliberately
+uses the short-operation policy: 100 warmups and 20 invocations per sample. The
+two write cases use five warmups and one invocation per sample. Other JVM cases
+use the short-operation policy too. The protocol records ten samples per case.
+
+Android defaults to smoke when no instrumentation argument is supplied. A green
+provider result alone is insufficient: use `-e mode benchmark` and validate the
+returned JSONL with `validate.py android OUTPUT.jsonl`. The validator rejects
+smoke-only, incomplete, duplicate and inconsistent timing evidence. Android uses
+three warmups; this does not establish ART compilation/thermal stationarity.
+
+The Kotlin harnesses fail on action or teardown errors. A second teardown failure
+can obscure the original exception, but cannot turn failure into a valid sample.
+Negative cache validation occurs in the binding before opening the file; the
+pinned shim's `options.to_core()?` precedes `Minigraf::open_with_options`.
+
+For the six common query/callback cases, result decoding and structural checks
+are part of the named foreign round trip. Expected-value comparisons, setup and
+reporting are outside the clock. The extra lifecycle/storage cases time their
+named calls: transaction-result JSON validation and reopened queries are outside
+the timer, as are the queries that validate preparation, registration and opening.
+Validate JSONL completeness and verify the packaged native artifact, not only a
+build-directory library, before accepting or publishing a run. Preserve failures
+and every sample; keep private machine/provider details out of public artifacts.
+
+## Swift/iOS
+
+Use the binding repository's `bench/device-farm/build.sh` and recorded core
+revision. GitHub builds the Release IPA/XCTest package and runs simulator oracle
+and smoke tests only. `testBenchmark` rejects simulator execution.
+
+For Device Farm `schedule-run`, set `test.type` to `XCTEST` and put the method
+selection in **`test.filter`**, not `test.parameters.filter`. Select
+`BenchmarkTests/testSmoke` before `BenchmarkTests/testBenchmark`. The first
+retained job used the latter, ineffective parameters field and ran all three
+methods; its logs show oracle and smoke passing before the benchmark in fresh
+app processes. This deviation is retained with that series.
+
+Set `configuration.customerArtifactPaths.iosPaths` to
+`["Documents/minigraf-bench-benchmark.jsonl"]` for timing (use `smoke` for smoke).
+The path is relative to the app container: a leading slash is rejected. In this
+series, Device Farm also failed to attach the relative-path file. The complete
+JSONL is therefore recovered from the retained automation log; each
+`MinigrafBench:` record is under 900 bytes. Do not assume file attachment succeeds.
+Extract one complete benchmark stream, not duplicated application/syslog copies,
+then run `validate.py swift-ios OUTPUT.jsonl`. Match the emitted build identity
+to the packaged `build.json` and retained checksums. A provider PASS or smoke
+stream alone does not validate timing evidence.
+
+The measured Swift harness is pinned by its binding revision and hash in
+`swift-ios-build.json`. Retain its package bytes; Device Farm re-signs the app
+for installation, so upload checksums identify the submitted artifacts.
@@ -0,0 +1,26 @@
+"""Checks cannot silently disappear under Python's optimization flag."""
+import pathlib
+import subprocess
+import sys
+import unittest
+
+HERE = pathlib.Path(__file__).resolve().parent
+
+
+class GuardrailTests(unittest.TestCase):
+    def test_optimized_driver_is_rejected(self):
+        result = subprocess.run([sys.executable, '-O', str(HERE / 'wasi.py'), '--help'],
+                                capture_output=True, text=True, timeout=10)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertIn('do not disable', result.stderr)
+
+    def test_optimized_evidence_validator_is_rejected(self):
+        evidence = HERE.parent.parent / 'docs/benchmarks/839-bindings/evidence/wasi-1.jsonl'
+        result = subprocess.run([sys.executable, '-O', str(HERE / 'validate.py'), 'wasi', str(evidence)],
+                                capture_output=True, text=True, timeout=10)
+        self.assertNotEqual(result.returncode, 0)
+        self.assertIn('do not disable', result.stderr)
+
+
+if __name__ == '__main__':
+    unittest.main()
@@ -0,0 +1,28 @@
+import unittest
+
+from wasi import check_query_output, check_process
+
+
+class ReplOracleTests(unittest.TestCase):
+    def test_accepts_unordered_complete_rows(self):
+        check_query_output('?n\n--------------------\n3\n1\n0\n2\n\n4 result(s) found.\n', range(4))
+
+    def test_rejects_empty_success(self):
+        with self.assertRaises(AssertionError):
+            check_query_output('No results found.\n', range(4))
+
+    def test_rejects_wrong_rows_with_equal_count_and_sum(self):
+        with self.assertRaises(AssertionError):
+            check_query_output('?n\n--------------------\n0\n0\n3\n3\n\n4 result(s) found.\n', range(4))
+
+    def test_rejects_error_even_with_zero_exit(self):
+        with self.assertRaises(AssertionError):
+            check_process(0, 'Error: query failed\n')
+
+    def test_rejects_nonzero_exit(self):
+        with self.assertRaises(AssertionError):
+            check_process(1, '')
+
+
+if __name__ == '__main__':
+    unittest.main()
@@ -0,0 +1,56 @@
+"""Validate a complete #839 timing JSONL; a passing smoke is not timing evidence."""
+import collections
+import json
+import math
+import pathlib
+import sys
+
+if not __debug__:
+    raise RuntimeError('do not disable validation with Python -O')
+
+COMMON = {
+    'prepared_execute/1k', 'cursor_batch_1/1k', 'cursor_batch_256/1k',
+    'set_roundtrip/2', 'predicate/1k', 'aggregate/1k', 'prepare/point',
+    'register_predicate/first', 'register_aggregate/first', 'fact_log_batch_256/1k',
+    'options_error/negative_cache', 'options_readonly_open/1k',
+    'file_transact/1k', 'log_writer_append_finish/1k',
+}
+WASI = {'repl_startup_empty', 'repl_script/1k', 'repl_query_roundtrip/1k'}
+
+
+def validate(records, runtime):
+    assert runtime in ('jvm', 'android', 'swift-ios', 'wasi'), 'unknown runtime'
+    expected = {f'binding/{runtime}/{name}' for name in (WASI if runtime == 'wasi' else COMMON)}
+    assert all(r.get('type') in ('environment', 'sample', 'checked', 'complete') for r in records)
+    environments = [r for r in records if r['type'] == 'environment']
+    complete = [r for r in records if r['type'] == 'complete']
+    checked = [r['id'] for r in records if r['type'] == 'checked']
+    assert len(environments) == len(complete) == 1, 'missing or duplicate run metadata'
+    assert environments[0]['mode'] == complete[0]['mode'] == 'benchmark', 'not a timing run'
+    assert complete[0]['count'] == len(expected)
+    assert len(complete[0]['ids']) == len(expected) and set(complete[0]['ids']) == expected
+    assert len(checked) == len(expected) and set(checked) == expected, 'incomplete checks'
+    groups = collections.defaultdict(list)
+    for record in records:
+        if record['type'] == 'sample':
+            groups[record['id']].append(record)
+    assert set(groups) == expected, 'missing or unexpected sample IDs'
+    for name, samples in groups.items():
+        assert len(samples) == 10 and {s['sample'] for s in samples} == set(range(10))
+        if runtime == 'wasi':
+            iterations = 20 if name.endswith('repl_query_roundtrip/1k') else 1
+        else:
+            iterations = 1 if name.endswith(('file_transact/1k', 'log_writer_append_finish/1k')) else 20
+        for sample in samples:
+            assert sample['iterations'] == iterations
+            assert type(sample['total_ns']) is int and sample['total_ns'] > 0
+            assert math.isfinite(sample['ns_per_op']) and sample['ns_per_op'] > 0
+            assert math.isclose(sample['ns_per_op'], sample['total_ns'] / iterations, rel_tol=1e-12)
+    return len(expected), sum(map(len, groups.values()))
+
+
+if __name__ == '__main__':
+    runtime, filename = sys.argv[1:]
+    records = [json.loads(line) for line in pathlib.Path(filename).read_text(encoding='utf-8').splitlines()]
+    cases, samples = validate(records, runtime)
+    print(f'Validated {cases} cases and {samples} timing samples for {runtime}')
@@ -0,0 +1,142 @@
+"""Actual WASI REPL timings; host framing/pipe overhead is included explicitly."""
+import argparse
+import hashlib
+import json
+import pathlib
+import queue
+import re
+import subprocess
+import threading
+import time
+
+if not __debug__:
+    raise RuntimeError('do not disable Python validation with -O')
+
+
+def check_process(status, stderr):
+    assert status == 0, 'WASI process failed'
+    assert not stderr.strip(), 'WASI process reported an error'
+
+
+def check_query_output(output, expected):
+    lines = output.splitlines()
+    if lines and re.fullmatch(r'✓ Transacted successfully \(tx: \d+\)', lines[0]):
+        lines = lines[1:]
+    assert lines[:2] == ['?n', '-' * 20], 'missing or wrong result header'
+    assert lines[-2] == '', 'missing result terminator'
+    expected = list(expected)
+    assert lines[-1] == f'{len(expected)} result(s) found.', 'wrong reported count'
+    rows = [int(line) for line in lines[2:-2]]
+    assert len(rows) == len(expected), 'wrong row count'
+    assert sum(rows) == sum(expected), 'wrong row sum'
+    assert sorted(rows) == sorted(expected), 'wrong row values'
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--runtime', type=pathlib.Path, required=True)
+    parser.add_argument('--wasm', type=pathlib.Path, required=True)
+    parser.add_argument('--core-revision', required=True)
+    parser.add_argument('--output', type=pathlib.Path, required=True)
+    parser.add_argument('--mode', choices=['smoke', 'benchmark'], default='smoke')
+    args = parser.parse_args()
+    args.output.mkdir(parents=True, exist_ok=False)
+    samples_file = args.output / 'samples.jsonl'
+    def emit(record):
+        with samples_file.open('a') as stream:
+            stream.write(json.dumps(record) + '\n')
+    flags = ['run', '-C', 'cache=n', '-C', 'parallel-compilation=n']
+    command = [str(args.runtime.resolve()), *flags, str(args.wasm.resolve())]
+    sha = lambda path: hashlib.sha256(path.read_bytes()).hexdigest()
+    emit(dict(type='environment', mode=args.mode, core_revision=args.core_revision,
+              wasm_sha256=sha(args.wasm), runtime_sha256=sha(args.runtime),
+              runtime_version=subprocess.check_output([str(args.runtime), '--version'], text=True).strip(),
+              runtime_flags=flags, storage='in-memory; no preopened directories',
+              runtime_cache=False, host_driver='Python subprocess pipes; framing overhead included',
+              warmups=3, samples=10, rss_verified=False))
+    transaction = '(transact [' + ' '.join(f'[:bench/e{i} :bench/n {i}]' for i in range(1000)) + '])\n'
+    query = '(query [:find ?n :where [?e :bench/n ?n]])\n'
+    expected = list(range(1000))
+    (args.output / 'fixtur
10BC8
e.datalog').write_text(transaction + query)
+    checked = []
+    def measure(name, action, validate, iterations=1):
+        case = 'binding/wasi/' + name
+        def invoke():
+            start = time.perf_counter_ns()
+            value = action()
+            elapsed = time.perf_counter_ns() - start
+            validate(value)
+            return elapsed
+        if args.mode == 'benchmark':
+            for _ in range(3):
+                invoke()
+            for sample in range(10):
+                total = sum(invoke() for _ in range(iterations))
+                emit(dict(type='sample', id=case, sample=sample, iterations=iterations,
+                          total_ns=total, ns_per_op=total / iterations))
+        else:
+            invoke()
+        checked.append(case)
+        emit(dict(type='checked', id=case))
+    def process(script):
+        return subprocess.run(command, input=script, text=True, capture_output=True, timeout=60)
+    def empty_valid(result):
+        check_process(result.returncode, result.stderr)
+        assert result.stdout == '', 'unexpected empty-input output'
+    def script_valid(result):
+        check_process(result.returncode, result.stderr)
+        check_query_output(result.stdout, expected)
+    measure('repl_startup_empty', lambda: process(''), empty_valid)
+    measure('repl_script/1k', lambda: process(transaction + query), script_valid)
+
+    # Start/compile/seed outside timing; validate every timed response before
+    # accepting its sample. No inferred engine-only latency by subtraction.
+    with (args.output / 'persistent-stderr.txt').open('w+') as errors:
+        proc = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+                                stderr=errors, text=True, bufsize=1)
+        lines = queue.Queue()
+        def reader():
+            for line in proc.stdout:
+                lines.put(line)
+            lines.put(None)
+        thread = threading.Thread(target=reader, daemon=True)
+        thread.start()
+        def get_line():
+            line = lines.get(timeout=30)
+            assert line is not None, 'REPL exited before completing its response'
+            return line
+        def query_once():
+            proc.stdin.write(query)
+            proc.stdin.flush()
+            response = []
+            while True:
+                line = get_line()
+                response.append(line)
+                if re.fullmatch(r'\d+ result\(s\) found\.\n', line):
+                    return ''.join(response)
+        try:
+            proc.stdin.write(transaction)
+            proc.stdin.flush()
+            assert re.fullmatch(r'✓ Transacted successfully \(tx: \d+\)\n', get_line())
+            check_query_output(query_once(), expected)
+            measure('repl_query_roundtrip/1k', query_once,
+                    lambda output: check_query_output(output, expected), iterations=20)
+            proc.stdin.close()
+            proc.wait(timeout=30)
+            thread.join(timeout=5)
+            errors.flush()
+            errors.seek(0)
+            check_process(proc.returncode, errors.read())
+            assert lines.get(timeout=5) is None, 'unexpected trailing REPL output'
+        finally:
+            if proc.poll() is None:
+                proc.kill()
+                proc.wait()
+            proc.stdout.close()
+    assert len(checked) == 3
+    emit(dict(type='complete', mode=args.mode, ids=checked, count=len(checked)))
+    print('Validated', len(checked), 'WASI cases in', args.mode, 'mode')
+
+
+if __name__ == '__main__':
+    main()
@@ -1,5 +1,10 @@
 # Binding benchmarks (#776 P16)
 
+The remaining manual work is tracked in [#839's scope and coverage plan](benchmarks/839-bindings/README.md).
+For that issue, new timings come only from Device Farm and the dedicated Linux
+bench; macOS timing is excluded. GitHub may build and check correctness, and the
+existing nightly measurements described here continue unchanged.
+
 These are foreign-runtime measurements, independent of native Criterion data.
 The `bindings` matrix in `.github/workflows/bench.yml` selects the whole fixed
 shortlist in each binding's `bench/nightly.*` entry point. Core and binding PRs
@@ -73,14 +78,16 @@ performed locally. These latency cases do not enforce RSS bounds.
 
 ## Deliberate dispositions requiring release sign-off
 
-- Java/JVM: no JVM latency history yet. Python exercises the shared UniFFI
-  adapter, but does not measure JVM allocation, JNI or JVM callbacks. JVM
-  timings remain a manual release check, not claimed nightly coverage.
+- Java/JVM: no JVM nightly latency history. Python exercises the shared UniFFI
+  adapter, but does not measure JVM allocation, JNA or JVM callbacks. The
+  [#839 manual series](benchmarks/839-bindings/results.md) supplies separate
+  release evidence; acceptance remains pending.
 - Android: ART/ABI and device storage timing require a fixed physical device;
   emulator timings on a hosted Linux runner are not a substitute. Manual
   device benchmarks and release sign-off remain required.
 - Swift: ARC/callback and Apple-device storage timing require stable Apple
-  hardware. Manual macOS/iOS benchmarks and release sign-off remain required;
+  hardware. Manual iOS benchmarks and release sign-off remain required for #839;
+  macOS timing is explicitly excluded from that issue's approved scope.
   Linux UniFFI tests do not provide these timings.
 - WASI: browser Wasm does not measure the WASI runtime. Tier 2 WASI REPL startup
   and runtime overhead remain manual; it exposes no matching foreign prepared