Reproducible benchmarking scripts for the OpenHuman Rust core as an embedded
library (no RPC server), built around the library-profile and rss-bench
binaries (see src/bin/library_profile/main.rs). Full write-up:
docs/library-benchmarking.md. Prior
findings: docs/resource-profiling-session-2026-07-21.md.
Five driver scripts: library-bench.sh (per-scenario RSS/duration),
library-cpu.sh (samply), library-heap.sh (dhat), library-fleet.sh
(fleet-scale sweep + budget gate), and library-instances.sh (multi-process
instance sweep).
Builds library-profile + rss-bench, runs each scenario N times as a fresh
process, and aggregates median/min/max duration, settled RSS, retained delta,
and peak delta into summary.json + summary.md.
./scripts/profile/library-bench.sh # all 7 scenarios, default build, 5 repeats
./scripts/profile/library-bench.sh --slim --repeat 7 # slim (no-default-features) build
./scripts/profile/library-bench.sh --scenarios "long-agent,subagents" --turns 50 --warmResults land in target/profile/rust-library/bench-<timestamp>/ (or --out DIR).
Wraps samply record around one scenario, isolated from persistence/timezone
noise by default (matching the documented cold-path CPU recipe).
./scripts/profile/library-cpu.sh subagents
./scripts/profile/library-cpu.sh long-agent -- OPENHUMAN_PROFILE_TURNS=50
samply load target/profile/rust-library/subagents-cpu.json.gzBuilds the rss-bench-dhat variant and runs one scenario under dhat. RSS and
timing numbers from this build are perturbed by instrumentation; use it only
for allocation-site/retained-bytes attribution, not for RSS comparisons.
./scripts/profile/library-heap.sh memory-ingest
# open https://nnethercote.github.io/dh_view/dh_view.html and load
# target/profile/rust-library/dhat-memory-ingest.jsonBuilds library-profile + rss-bench, sweeps the fleet scenario (N
concurrent live agents with latency-realistic mock inference) across a list
of agent counts, aggregates medians per N, and gates on whether the
projected footprint at the target agent count fits the RAM budget.
./scripts/profile/library-fleet.sh --agents 100 --latency-ms 200
./scripts/profile/library-fleet.sh --agents "50,100,500" --target 1000 --budget-mib 2048Results land in target/profile/rust-library/fleet-<timestamp>/ (or
--out DIR). Exits nonzero if any swept N reports fits: false (use
--no-gate to report only). See
docs/library-benchmarking.md
for the budget math.
Spawns N independent library-profile processes (each a live instance held
alive via OPENHUMAN_PROFILE_HOLD_SECS), staggered on startup, and measures
per-process cost and box survivability — the opencompany "N independent
processes/containers" deployment model, as opposed to library-fleet.sh's
"N agents in one process" model. Samples aggregate sum-RSS + live count every
2s while instances hold, captures a vm_stat snapshot at peak (and a
best-effort footprint sample if that macOS tool is present), then
aggregates per swept N: launched/ok counts, median settled RSS per instance,
mean and peak aggregate sum-RSS, and — on Linux, where it's meaningful —
summed PSS.
./scripts/profile/library-instances.sh --instances "10,50" --hold-secs 30
./scripts/profile/library-instances.sh --instances "100,500" --max-instances 500 --gateResults land in target/profile/rust-library/instances-<timestamp>/ (or
--out DIR). Refuses to spawn more than --max-instances (default 200)
without an explicit raise — see the script's --help for the RAM math. Exits
nonzero with --gate if any instance failed to complete cleanly (nonzero
exit or missing/invalid JSON); default is report-only. See
docs/library-benchmarking.md
for the fleet-vs-instances framing.
Drives the skill-run scenario with K parallel skill runs and asserts the DoD:
with the shared runtime pool ON, the process tree grows by ~one pooled worker,
not K interpreters. The scenario hard-asserts tree.child_count <= max_workers
for K > 1 (nonzero exit); this script runs it and reports a pooled-vs-unpooled
comparison. A regression that reintroduces per-run forking fails the gate.
./scripts/profile/library-pool-gate.sh # K=8, max_workers=1
./scripts/profile/library-pool-gate.sh --concurrency 16 --skip-buildExits 0 = pass, 1 = regression. SKIPs (exit 0) when no system node is on
PATH (the scenario must never download an interpreter), so node-less CI
runners don't false-fail.
# 1. Baseline RSS/duration across all scenarios
./scripts/profile/library-bench.sh
# 2. CPU attribution for the slowest/most interesting scenario
./scripts/profile/library-cpu.sh subagents
# 3. If a scenario's RSS looks off, drill into live heap
./scripts/profile/library-heap.sh subagentsAll scripts require jq for JSON parsing/aggregation; library-cpu.sh also
requires samply (cargo install samply). Build commands use
GGML_NATIVE=OFF to work around the Apple Silicon whisper-rs/llama.cpp NEON
fp16 build issue.