Notable changes per release. forkd follows Semantic Versioning once it reaches 1.0; until then, the minor version can break compatibility.
Spawned sandboxes can now back their guest RAM with 2 MiB hugepages
(MFD_HUGETLB | MFD_HUGE_2MB) instead of 4 KiB pages, reducing TLB
pressure on spawn-many and live-BRANCH bulk-copy. Opt in per spawn via
--hugepages (CLI), hugepages: true (REST / SDK). The flag requires
--live-fork — only the v0.4 memfd path benefits; file-backed restore
is unaffected.
Plumbed end-to-end by external contributor @theflashwin in
#230 — their first PR to
the project, and unusually complete: Rust MemoryBackend::MemfdShared
extended to { use_hugepages: bool }, REST + CLI + Python SDK + TS SDK
all carry the flag, a forkd doctor check (HugePages_Free) surfaces
pool exhaustion before spawn, ENOMEM falls back to normal pages with a
warning, and bench/live-fork-pause-window/bench-hugepages.py
interleaves baseline / hugepages iterations for fair A/B numbers.
The bench harness ships with the PR but N=100 spawn numbers are still pending on a clean host — tracking in #233. PR README has @theflashwin's N=4 measurements from his hardware.
Drops the unmaintained rustls-pemfile dependency flagged by
RUSTSEC-2025-0134. The daemon's TLS path now goes through
rustls-pki-types directly. Internal-only change; HTTPS bind / cert
loading on the daemon are byte-compatible. (#232)
recipes/README.md now leads with a "by problem you're solving" table
that surfaces CI test parallelization, database fixture forking, and
browser farming alongside the AI-agent use case, instead of presenting
forkd as AI-only.
New recipe recipes/ci-parallel-pytest ships a complete Python
pytest fan-out demo (5 test files, ~30 tests across numpy / pandas /
sklearn / regex / stdlib) backed by a forkd snapshot so each worker
starts hot — import numpy etc. is already paid for in the parent.
Bench numbers from the dev box: ~20 ms per worker batch-spawned
vs ~2-3 s per cold container. (#231)
forkd's pre-v0.5.1 vmlinux was Linux 4.14.174 built 2021-07-14.
That kernel predated CONFIG_HW_RANDOM_VIRTIO, CONFIG_RANDOM_TRUST_CPU,
the random.trust_cpu= cmdline option (Linux 4.19+), and CONFIG_VMGENID
(Linux 5.20+). Symptom: getrandom(2) blocked forever inside a
restored guest because the CRNG never marked itself initialized —
anything that touched OpenSSL (pip install, urllib.request HTTPS,
requests) hung indefinitely.
Replacement: Linux 6.1.141 from Firecracker's CI bucket
(firecracker-ci/v1.13/x86_64/vmlinux-6.1.141, sha256
b36a4a1b10f33b9cfdcde3d1a787d9c090556a3edb211cd06d1f3f9a6c7e8724).
Verified end-to-end on the dev box on 2026-06-05:
| before | after | |
|---|---|---|
uname -r |
4.14.174 | 6.1.141 |
/dev/hwrng |
absent | present (virtio-rng) |
getrandom(GRND_NONBLOCK) |
EAGAIN | returns 16 bytes immediately |
ssl.create_default_context() |
blocks forever | 0.01 s |
urllib.request.urlopen("https://pypi.org/…") |
blocks forever | 0.25 s |
pip install numpy==2.0.2 |
exit 1, empty output | 21 s, exit 0 |
| VMGENID on FC restore | n/a | dmesg: random: crng reseeded due to virtual machine fork |
Two new helper scripts:
scripts/install-guest-kernel.sh— downloads the FC CI vmlinux into/var/lib/forkd/kernels/vmlinux; the recommended path.scripts/build-guest-kernel.sh— clones Linux 6.1 LTS, applies FC's reference microvm config, builds vmlinux from source (~10 min on a 20-core box). Use this when you want to audit / tune the kernel config or target a non-x86_64 architecture.
Migration: existing snapshot vmstates produced under 4.14 still restore on 6.1 — they just don't gain the new entropy device until they're re-baked. Run forkd from-image <image> --tag <tag> for each snapshot you care about to pick up CRNG init + /dev/hwrng + VMGENID.
forkd doctor's "kernel image" check now suggests running scripts/install-guest-kernel.sh when no vmlinux is found.
Snapshots can now record a parent_tag + content-hash edge to an
earlier snapshot, letting agents stack incremental layers on a
warmed base (base → +numpy → +pandas → +sklearn) instead of
producing a flat 1.5-GiB-each snapshot per dependency set. The
daemon walks the chain at spawn time and assembles the memory image
in one pass; correctness verified across depths 1-3 by the Phase 5
bench (bench/chain-spawn/RESULTS-v0.5.md)
with 90/90 (100%) probe passes. Tracking issue
#216.
- Phase 1 (PR #213) —
forkd_vmm::chainlibrary:resolve_chain,verify_parent_hashes,assemble_chain_memorywith a reflink- preferredcp(base) + apply_diffwalk. 7 unit tests cover cycle detection, missing-parent actionable errors, and hash mismatch. - Phase 2a (PR #214) — chain-aware
POST /v1/sandboxes: when the target tag hasparent_tagset, the controller transparently walks the chain, verifies each link'sparent_content_hash, assembles memory into a per-spawn temp file, and FC-restores from that. Pre-existing flat-snapshot spawns unchanged. - Phase 2b (PR #215) —
forkd snapshot-diff --from --tag --execCLI verb. Spawns a one-shot sandbox from the base, runs the exec in-guest, then BRANCHes withparent_tagset so the resulting snapshot records the chain edge. The daemon's BRANCH endpoint acceptsparent_tagon the request body (validated against the source's snapshot tag). - Phase 5 (PR #217) — chain-spawn bench + a Phase 2a bug fix.
Headline: on a 512 MiB base (ext4), a depth-3 chain spawn took
p50 = 1668 ms vs 746 ms for the flat-equivalent. The ~460 ms
per-link tax tracks SHA-256 of the base — the mmap-once
incremental verify optimization is queued for v0.6. Same PR also
fixed a real bug where the chain-assembled
memory.binwas unlinked byrestore_many_with's startup sweep before FC could load it (caught by running the bench, never by unit tests). New regression testwork_dir_sweep_preserves_chainstage_subdirectory. - Phase 4 (PR #219) — chain-aware management verbs:
forkd snapshot-info <tag>(GET /v1/snapshots/:tag/info) — 8000 chain depth, ancestors, dependents, on-disk sizes.forkd rmi <tag> [--cascade | --force]— pre-Phase-4rmisilently orphaned chained children; now refuses with HTTP 409 + actionable message until the caller picks an escape hatch.forkd snapshot-compact --from <tag> --to <new>(POST /v1/snapshots/:tag/compact) — materialize a chain into a new flat snapshot with noparent_tag. Stages via.compact-staging-<to>/+rename(2)so a mid-compact crash leaves nothing half-written.
- Phase 3 (PR #220) — Hub pack/unpack bumped to manifest v2.
When the head has
parent_tag.is_some(),forkd packwalks the chain on disk and bundles every ancestor under per-link tar prefixes;unpackmaterializes one snapshot directory per link. Depth-0 (base) snapshots keep emitting v1 packs so older clients keep working. Unsafechain[].tagentries (path traversal) are rejected at manifest-parse time before any file body is extracted. - Phase 6 — docs (this changelog entry, README v0.5 section,
docs/API.mdSnapshotInfoDetail+ new endpoints,docs/HUB.mdpack v2 spec,DESIGN-v0.5-diff-snapshot-chains.md).
The v0.4 live-fork path — BRANCH that pauses the source for sub-50 ms instead of the 150-300 ms a Diff BRANCH costs — is now reachable from every supported entry point. The underlying mechanism (UFFD_WP to capture dirty pages out-of-band, memfd-backed RAM so Firecracker and the controller share the same page cache) was prototyped in Phase 6 (PRs #194-#202). This release wires it through the public surface:
- REST (PR #204) —
POST /v1/sandboxes/:id/branchaccepts a canonicalmode: "full" | "diff" | "live"field. The legacydiff: trueboolean still works; setting both yields HTTP 400. Newwait: falsetoggles fire-and-forget (return after the source resumes, ~10 ms; snapshot reachesstatus: "ready"asynchronously).POST /v1/sandboxesacceptslive_fork: trueso the spawned sandbox boots with memfd-backed RAM — a prerequisite for latermode: "live"BRANCHes from it. - CLI (PR #205) —
forkd snapshot --from-sandbox <id> --liveand--live --no-wait. Clap enforces--diff/--livemutex and--no-waitrequires--live. Gap:forkd fork --live-fork(spawn-time opt-in) isn't surfaced on the CLI yet — use the SDK orPOST /v1/sandboxesdirectly to spawn a live-fork parent. Tracking as a follow-up. - SDKs (PR #206) — Python (
Controller.branch_sandbox(mode=, wait=),Controller.spawn_sandboxes(live_fork=)), TypeScript (branchSandbox({ mode, wait }),spawnSandboxes({ liveFork }),BranchModetype re-exported), MCP (branch_sandbox/spawn_sandboxestools). All backwards-compatible: existingdiff=Truecallers keep working and still drive v0.3.x daemons unchanged. - Doctor (PR #207) — two new checks (
uffd_wp (v0.4 live BRANCH)andmemfd_create (v0.4 live BRANCH)) probe the kernel prerequisites. WARN-not-FAIL: v0.3 Diff/Full still work without these. Smart hints — EPERM maps tosysctl vm.unprivileged_userfaultfd=1, ENOSYS to "kernel < 5.7, Diff still works".
Prereqs for mode: "live": Linux ≥ 5.7 (UFFD_WP),
unprivileged_userfaultfd=1 or CAP_SYS_PTRACE, the vendored
Firecracker fork from
deeplethe/firecracker:forkd-v0.4-mem-backend-shared-v1.12
— upstream FC doesn't yet ship mem_backend.shared = true. See
docs/VENDORED-FIRECRACKER.md.
Bench numbers (bench/live-fork-pause-window/RESULTS-v0.4.md)
on a clean Hub-pulled python-numpy source (1.5 GiB, Intel i7-12700,
ext4 on HDD):
| mode | pause p50 | pause p90 | RT p50 |
|---|---|---|---|
| live-sync | 56 ms | 64 ms | 13 730 ms |
| live-async | 54 ms | 241 ms | 69 ms |
| diff | 202 ms | 418 ms | 13 461 ms |
| full | 13 550 ms | 14 268 ms | 13 559 ms |
Headline: 3.6× faster pause vs v0.3 Diff at p50, and the gap
widens on slower storage because Live's pause is disk-independent.
wait=false gives callers a ~70 ms HTTP return (vs 13.7 s for sync),
~200× RT improvement for fire-and-forget BRANCH.
crates/forkd-controller/src/auth.rs::constant_time_eq was advertised
as constant-time but leaked the presented token's length via two
distinct paths:
- The length-mismatch branch's "fake work" loop used
x.wrapping_mul(0)— the compiler is allowed to (and LLVM does) delete the loop as dead code, so response time was monotonic in the longer slice's length. - Even if the loop had been preserved, taking different branches for length-equal vs length-mismatch was itself a timing oracle.
For a deployment with a fixed-length token (e.g. 32-byte hex from
forkd doctor), the length oracle collapses security to a single
attempted length, after which the equal-length branch is real
constant-time.
Fix. Replaced with a subtle::ConstantTimeEq call against a
zero-padded copy of the presented token. Same code path regardless
of input length; length difference is folded into the result via a
non-short-circuiting bitwise AND. subtle was already an indirect
dep via the rustls / aws-lc-rs subtree, so the patch adds no new
crate to the lockfile.
Thanks to @m-dmupba for the report and the precise repro.
The "BRANCH 3-5 pause_ms jumps to 1.3-1.5 s on the same source" anomaly documented in v0.3 results is gone. Root cause: ext4's delayed allocation + multi-block allocator + writeback throttle + block-bitmap checksumming compounded per BRANCH; each BRANCH's 500 MiB+ memory.bin write triggered increasing ext4 metadata work.
Fix: posix_fallocate the destination memory.bin to its full size
right after mkdir snap_dir, before the diff-mode background cp or
Firecracker's /snapshot/create write. ext4 reserves the extents
up-front; subsequent writes don't run ext4_mb_new_blocks or update
block bitmaps in-band.
Measured impact (ext4 SSD, coding-agent-fork-prewarm-v1 source,
10 consecutive diff BRANCHes):
| BRANCH | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| before | 350 | 250 | 1300 | 1400 | 1500 | 2700 | 1500 | 1800 | 2700 | 1500 |
| after | 585 | 286 | 344 | 161 | 369 | 153 | 189 | 162 | 324 | 174 |
- BRANCH 6: 17.6× faster (2700 ms → 153 ms)
- Median across BRANCH 3-10: ~1700 ms → ~200 ms (~8.5×)
- After-curve matches a tmpfs control to within noise
Best-effort: on filesystems that don't support posix_fallocate
(tmpfs, NFS, FAT, etc.) the call returns ENOSYS, a WARN is logged,
and the BRANCH continues with the previous behavior.
Five rounds of probing produced the diagnosis. Full investigation:
bench/pause-window/PROBE-multi-branch-anomaly.md.
forkd doctorextended to 14 checks (PR #142 from v0.3.3 — listed there but worth re-mentioning since it shipped late in the cycle): hw-virt, FC version, docker, snapshot-dir space added.- CI now auto-dispatches
publish-pypi.ymlfromrelease.ymlvia explicitworkflow_dispatch(PR #144). v0.3.1-v0.3.3 all required manualgh workflow run; v0.3.4 is the first that should auto-publish. forkd snapshot --from-sandbox,forkd ls,forkd kill,forkd rmi,forkd from-image,forkd bench,forkd doctorall surface abranch_countwarning when the source has been BRANCHed ≥3 times (less relevant now that the anomaly is fixed, but kept for diagnostic clarity).
The forkd binary gained a developer-experience cluster:
forkd doctor— 10 host-readiness checks (KVM, tap, netns, firecracker binary, kernel image, daemon, ...) with PASS / WARN / FAIL / SKIP per check and a one-line fix hint for each failure. Safe to run unprivileged; skips root-only checks with a note. Use this first after a freshscripts/setup-host.sh.forkd bench— a representative spawn → exec → branch(diff=true) → fanout → cleanup cycle against the live daemon. Screenshot-friendly per-step timing. Answers "is forkd actually fast on this box?" in one command.forkd from-image <docker-image> --tag <tag>— Docker pull → ext4 → boot + warmup → pause → register tag, in one verb. boxlite parity for "docker pull Xand you're done".forkd ls— list live sandboxes the daemon knows about. Table output (id / snapshot / pid / netns / guest_addr).forkd kill <id>.../--all/--tag <tag>— terminate sandboxes via DELETE /v1/sandboxes/:id without hand-writing curl.forkd rmi <tag>...— delete snapshot tags (docker-style). Tries daemon DELETE first; falls back to direct disk removal when the daemon is unreachable or didn't know the tag.
forkd images output also got a table refresh: new MEMORY and
CREATED columns (relative age), most-recent-first sort, snapshot
count + total bytes footer.
The CLI now exposes the v0.3 Diff BRANCH path:
forkd snapshot --from-sandbox sb-abc-0000 --diff --tag base-plus-pip
# ~200 ms pause, vs multi-second Full modeCloses the last gap from #28 — REST and both SDKs already exposed
diff; the CLI was the only one missing.
Host-side Python scripts (no rootfs build required) showing how to plug forkd into:
recipes/mcp-agent/— Claude Desktop / Cursor / Cline via MCPrecipes/crewai-fanout/— N CrewAI agents on N microVMsrecipes/autogen-branch/— AutoGenCodeExecutor+ mid-conversation BRANCHrecipes/openai-swarm/— Swarm/Agents handoff = BRANCHrecipes/speculative-agent/— the headline demo: BRANCH + N strategies + judge picks best. Tweet-friendly artifact (2595× faster than slowest strategy in the included example).
Each ships with a --dry-run mode that exercises the forkd plumbing
without an LLM key.
Follow-up to v0.3.1's "BRANCH 3-5 anomaly" finding. Two new probe
scripts and a refined attribution: FC is off-CPU ~94 % of the slow
BRANCH window; the dominant contributors are userspace futex
contention (17/250 in-kernel-sleep samples), ext4 journal IO (~2 %),
and un-symbolized user-space CPU on the snapshot worker thread (FC
static-pie release has no frame pointers; needs DWARF or a debug
rebuild to drill further). See
bench/pause-window/PROBE-multi-branch-anomaly.md.
Direct consequence: original #118 Phase 2/3 scope needs revision.
- Quick start in both
README.mdandREADME-zh.mdrewritten aroundforkd doctor+forkd from-image+forkd bench— the modern user path that didn't exist before this release. @deeplethe/forkd0.3.1 published to npm (first npm release for the TS SDK).
Python SDK only. Closes the surface-parity gap between the REST API, the TypeScript SDK, and the Python SDK:
Controller.spawn_sandboxes(prewarm=...)— opt into the v0.2 prewarm path that amortizes first-BRANCH cold-cache cost.Controller.branch_sandbox(diff=..., measure_diff=...)— opt into v0.3 Diff BRANCH (and the measurement-only Diff sidecar) from Python. REST and TS SDK already had these.
No Rust changes; the workspace stayed at 0.3.1.
Lifts the v0.3.0 first-BRANCH-only restriction. The daemon now tracks
SandboxInfo.last_branch_memory_path — set to whichever BRANCH most
recently completed (Full or Diff). On the next diff: true request:
- Chain head set AND file exists → use it as the cp source. By construction it's source's complete state at that BRANCH's pause time, which is exactly the base the next diff needs.
- Chain head set BUT file missing (user
DELETEd an intermediate BRANCH) → fall back tosource_tag/memory.binwith a logged warning. Lossy but doesn't crash. - Chain head unset (first BRANCH on sandbox) → use
source_tag/memory.binas before.
Zero extra storage (each BRANCH's output is already on disk; we just point at it). No background tasks. No separate shadow file.
The has_branched: bool flag stays in SandboxInfo as a diagnostic;
the daemon no longer 400s on it. The previously-shipped 400 error
message for second-and-later diff BRANCHes is gone.
Measurement: 3 trials × 5 consecutive diff: true BRANCHes on a
mem-2048 SSD source. All 15 succeed. Diff sizes stay <1.2 MB per
BRANCH (bitmap-clear semantics confirmed). Aggregate source downtime
across the 5 BRANCHes is ~4.7 s vs ~70 s if these had been Full —
14× pause-window reduction over a multi-BRANCH workflow. Raw
data in bench/pause-window/multi-branch-sweep.csv; new sweep script
bench/pause-window/sweep-multi-branch.sh.
Anomaly noted but not blocked-on: pause_ms jumps from ~280 ms at BRANCH 1-2 to ~1.3-1.5 s at BRANCH 3-5 despite the diff size staying small. Likely a KVM / firecracker control-plane accumulating cost; filed for follow-up. Still ~10× better than Full mode at every cell.
Headline: source-pause window for BRANCH drops 6-143× depending on workload.
Idle 4 GiB SSD source: 29 s → 205 ms = 143×.
Typical agent workload (30-300 MiB dirty footprint on 2 GiB source): 6-15×.
Crossover at ~65 % source RAM dirty. Honest curve and practical
guidance in
bench/pause-window/RESULTS-v0.3.md.
This release also bundles v0.2.5's prewarm fix (#100) — variance reduction for BRANCH pause on cold-cache hits, sandbox-creation trade-off — and the v0.3-cycle scaffolding for the deferred live-fork plan (#101, kept as honest record + revival starting point).
Vm::snapshot_diff_toinforkd-vmm— calls Firecracker/snapshot/createwithsnapshot_type: "Diff", returns aDiffSnapshotcarrying both logical and physical sizes (the latter = on-disk allocated bytes = the BRANCH's dirty footprint).apply_diff(diff_path, base_path)helper —SEEK_DATA/SEEK_HOLEwalk the diff's allocated extents, 1 MiB chunks copied onto the same offsets of the base file. Returns bytes copied. Linux-only; non-Linux builds bail rather than silently degrading.ForkOpts.enable_diff_snapshots: bool— required on/snapshot/loadfor the resulting VM to admit Diff snapshot/create calls. Default false (v0.2 callers preserve identical behavior); daemon'screate_sandboxflips it to true for all daemon-spawned sources.POST /v1/sandboxes/:id/branchgains"diff": bool. When true, the daemon parallelizes the source-tag memory.bin copy with the source running, takes a Diff snapshot during a ~200 ms pause, resumes the source, joins the copy, and merges the diff onto the pre-copied output. The user-visible pause is just the Diff window — source TCP connections, kvmclock, and timers see a ~200 ms gap instead of seconds. Total BRANCH API latency is unchanged on SSD (still bandwidth-bound on the cp); only source DOWNTIME shrinks.SandboxInfo.has_branched: bool+Registry::mark_branched()gate that rejects second-and-later"diff": trueBRANCHes with a clear 400. Firecracker clears the dirty bitmap on every snapshot/create, so a second Diff would silently miss pages dirtied before BRANCH 1. Multi-BRANCH diff support (per-sandbox shadow file) is deferred to v0.3.1+; forkd's canonical "spawn → BRANCH once → fan out N → discard source" workflow only ever takes one BRANCH per sandbox, so the restriction covers ~80% of use cases.SnapshotInfogainsdiff_ms,diff_physical_bytes,diff_logical_bytes— populated whendiff: trueormeasure_diff: truewas set on the BRANCH request.- Measurement:
bench/pause-window/RESULTS-v0.3.mdwith the full A/B (5 memory sizes × 3 trials × 2 modes × 2 backends = 60 trials). Phase 1a numbers (sidecar Diff inside the existing Full pause) match phase 1b numbers (realdiff: true) within measurement noise — architecture validated. Honest caveats: idle- source best case; 256 MiB on tmpfs is a wash (control-plane floor exceeds memcpy); first-BRANCH-only restriction. - Sweep scripts:
sweep-diff.sh(phase 1a sidecar timing) andsweep-diff-real.sh(phase 1b real A/B). Raw data CSVs checked in.
Deferred to v0.4+. Live-fork via memfd + uffd_wp is tracked in issue #101. The architecture has an open question on source-divergence sync that we haven't sketched concretely enough to commit to weeks of Firecracker maintenance for. v0.3 is now pursuing cheaper pause-window wins — diff snapshots, NVMe + io_uring, pre-emptive background snapshot — see
docs/ROADMAP.md. The scaffolding below stays in the repo because it's reusable when/if the project picks the live-fork work back up.
MemoryBackend::Userfaultenum variant inforkd-vmm, reserved for the (now-deferred) live-branching design. Setting it today errors out ofrestore_many_withwith a pointer todocs/design/userfaultfd.md; no caller can accidentally rely on a behavior we haven't built.forkd-uffdcrate, phase 1. New workspace member containing the library half of the userfaultfd page-fault handler. Implements Firecracker's UDS handshake:recvmsgwithSCM_RIGHTSto receive the uffd file descriptor + a JSON-encodedVec<GuestRegionUffdMapping>describing the host VAs of guest memory regions. Wire-compatible with Firecracker v1.10.1'ssrc/firecracker/examples/uffd/uffd_utils.rs. Ships aforkd-uffd-handlerbinary that accepts the handshake and exits — noUFFDIO_COPYevent loop. Round-trip handshake test paired oversocketpair(2)so CI exercises the parser without needing a real Firecracker. Reusable as-is if the live-fork plan revives; orthogonal value as a reference implementation of the Firecracker uffd protocol.firecracker-patch/directory — REMOVED. Originally drafted a ~100 LOC patch for aMemoryBackend::MemfdFirecracker extension (forking upstream at v1.10.1). After v0.3 phase 1 shipped 143× on vanilla Firecracker, we evaluated whether to actually take the fork path and decided not to: the memfd value-add doesn't add sharing capability we don't already have viammap MAP_PRIVATE, and the fork-maintenance cost (own CI, rebase on every upstream tag, track CVEs, weakened trust story) isn't justified for the remaining pause-window headroom. Reasoning indocs/design/userfaultfd.md§ "Why we won't fork Firecracker"; revival criteria in issue #101.
- Sandbox prewarm: amortize the cold-cache penalty at create time.
New
"prewarm": truefield onPOST /v1/sandboxes. When set, the daemon performs a throwaway snapshot to scratch storage (configurable, default/dev/shm/forkd-prewarm) immediately after each child is restored, faulting in all guest pages and populating KVM EPT. After prewarm, the first user-visible BRANCH runs at steady-state speed instead of paying the measured 2-9x cold-cache penalty (seebench/pause-window/RESULTS-v0.2.md). The cold cost is moved, not eliminated: sandbox creation pays one cold-pause-window worth of latency in exchange for a consistent BRANCH latency from the first call. Useful when BRANCH is the request handler with an SLO; not useful for create-then-one-BRANCH end-to-end latency. Off by default — opt in via the request body. Implemented inforkd-vmmasVm::prewarm()+ForkOpts::prewarm_scratch_dir; daemon adds--prewarm-scratch-dirflag /FORKD_PREWARM_SCRATCH_DIRenv var. - forkd Hub MVP.
forkd pull <owner>/<name>resolves through a registry.json published in this repo and downloads.forkd-snapshot.tar.zstpacks from GitHub Releases. sha256- verified, optional@<version>pinning, free public hosting, no per-package config. Full spec + publish workflow:docs/HUB.md. First published pack:deeplethe/langgraph-react(14.5 MiB after 35× zstd compression). - CLI
forkd pullrewrite to use the new registry indirection with sha256 integrity check, replacing the previous string-build approach againstforkd-hub.deeplethe.com(which we never set up the DNS for).
recipes/langgraph-react/— branch-and-fan-out demo of a real LangGraph-style ReAct agent. Source agent runs 2 ReAct steps with tool calls, BRANCH pauses it (~4 s on SATA SSD, ~160 ms on tmpfs), 3 grandchildren spawn with different steering hints, each produces a different itinerary inheriting the same prior cognitive state. Full writeup- asciinema cast embedded in README + first real-run artifacts
at
recipes/langgraph-react/results-2026-05-18/.
- asciinema cast embedded in README + first real-run artifacts
at
recipes/coding-agent-fork/— the "why not parallel prompt?" rebuttal. 50 MiB binary blob travels byte-identically across 4 sandboxes through a single BRANCH; three grandchildren each apply different fix strategies (sed / rewrite / skip-tests) and produce visibly different outcomes. Artifacts atrecipes/coding-agent-fork/results-2026-05-19/.recipes/cube-langgraph/(stub) — design sketch for CubeSandbox + forkd side-by-side deployment. Pairs withdocs/INTEGRATION-CUBESANDBOX.mdwhich compares the two projects honestly and proposes 3 concrete integration patterns.
- tmpfs
/tmpmount inforkd-init.sh. Per-VM 256 MiB tmpfs prevents the shared-rootfs corruption that hit the langgraph-react demo when 3 grandchildren wrote concurrently to the same on-disk inode. Affects every recipe built after this commit; no API change. Always put writable demo state under/tmp.
bench/pause-window/RESULTS-v0.2.md— first-cut measurement shows pause is storage-bound: 163 ms ± 7 ms on tmpfs (4 trials), 4.26 s ± 0.41 s on SATA SSD (5 trials) for the same 513 MiB source. Same forkd code; only--snapshot-rootdiffers. 5/5 connection survival, 0 in-flight loss across SSD trials. Surprising mechanism: in-guest agents are pause-blind because kvmclock's monotonic catch-up on resume races recv data delivery; the recv returns data before its timeout timer can fire. The userfaultfd bet's value sharpens to "external observers see the pause; in-guest agents barely notice".
create_sandboxsnapshot_tag validation gap (MEDIUM-HIGH, post-auth, fixed in PR #54).POST /v1/sandboxesacceptedreq.snapshot_tagfrom the request body and joined it directly intosnapshot_rootwithout callingis_safe_tag— unlike sister handlersdelete_snapshotandbranch_sandboxwhich both validated. The unvalidated tag also persisted intoSandboxInfo.snapshot_tagand later flowed intoread_snapshot_volumesduring BRANCH, whichserde_json::from_str'd the file at<snapshot_root>/<tag>/snapshot.json— letting an authenticated attacker control volume mounts of grandchild VMs. Full advisory: docs/SECURITY.md → 2026-05-17.- K8s manifest placeholder bearer token. The shipped
packaging/k8s/forkd-controller.yamlshipstoken: REPLACE_ME_*. Users who forget tosedit beforekubectl applywould get a daemon protected only by a publicly-known token. Fixed: daemon refuses to start if the token begins withREPLACE_ME/CHANGE_MEor is shorter than 16 bytes. boot_wait_secscap.POST /v1/snapshotspreviously accepted anyu64. Clamped to 60 s.
- BRANCH concurrency caps (PR #56). Two
POST /branchcalls on the same target tag now serialise via a per-tag in-flight set (second gets 409) — previously both could pass thevmstate.exists()TOCTOU and clobber each other. The daemon also admits at mostDEFAULT_BRANCH_CONCURRENCY(4) BRANCHes simultaneously; excess gets 503. Both bounds use an RAIIBranchSlotguard so every error path releases cleanly.
pause_mson BRANCH responses (PR #58).SnapshotInfonow carries an optionalpause_mspopulated bybranch_sandboxwith the measuredpause() → resume()envelope on the source VM. Also emitted as a structuredtracing::info!event. Powers the newbench/pause-window/harness.
- Pause-window harness + first-cut results. New
bench/pause-window/directory with a synthetic ping/pong agent, host-side echo server, orchestrator, and pure-function analyzer (14 unit tests, covered by a newbench-pythonCI job). 5 trials on real hardware (513 MiB source): mean pause 4.26 s ± 0.41 s, 0 in-flight loss, 5/5 connection survival. Surprising mechanism (in-guest agents are nearly pause-blind via kvmclock catch-up) documented inbench/pause-window/RESULTS-v0.2.md.
- Path traversal via
--tag(CVE-class, fixed).forkd snapshot,forkd unpack,forkd pull,forkd fork,forkd pack, andforkd pushaccepted arbitrary strings for--tagand used them inPath::join, which silently discards the base when the right side is absolute. A tag like/etc/forkd-bador../../etc/xcould write Firecracker snapshot files outside the data directory. The same risk extended to thetagfield ofmanifest.tomlinside a Snapshot Hub pack — a malicious or compromised pack could write its files anywhere the running user can write. Affects 0.1.0–0.1.2. Fixed by validating tags against[A-Za-z0-9_][A-Za-z0-9._-]{0,63}at every CLI surface and again on the manifest'stagfield. Full advisory: docs/SECURITY.md → Past advisories. forkd cleanupwould mis-classify live VMs as "safe to delete" becauselsofreturns empty stdout (only warnings on stderr) for Firecracker UNIX domain API sockets. Underforkd cleanup --yesthis would have torn down the work_dir of an actively running VM. Replaced the detection with a/proc/<pid>/fd/*readlink scan that explicitly checks whether any process holds an open handle inside the candidate directory.
forkd push <local-tag> <url>— HTTP PUT a packed snapshot to any URL (presigned PUT from R2/S3/etc. is the intended fit).forkd cleanup— sweep orphan/tmp/forkd-{fork,parent,unpack, pull}-*work directories left behind by crashed or killed runs. Dry-run by default;--yesto actually delete. Refuses to touch directories whose/procfd scan shows a live process.scripts/netns-teardown.sh— reversenetns-setup.sh. Dry-run by default, removes only^forkd-child-[0-9]+$netns. Docker bridges, system tap, andforkd-br0are untouchable without explicit--include-bridge/--include-tapflags.forkd snapshot --mem-size-mib— override the parent VM memory size (default 512 MiB). Required for memory-hungry warmup workloads; browser recipes need ≥ 2048 MiB to avoid Chromium OOM during snapshot.forkd snapshot --keep-workdir/forkd fork --keep-workdir— preserve/tmp/forkd-{parent,fork}-<tag>/after a successful run for post-mortem inspection. The default behaviour now removes the work_dir on success (failure paths still preserve it).- Pre-flight check on
forkd snapshot/forkd forkrefuses to start when another forkd run on the same tag is already in flight (live process holding sockets in the same work_dir), and cleans stale work_dirs from earlier crashes before proceeding. - Snapshot Hub: pack-and-go via
forkd pack/forkd unpack/forkd pull. Manifest records per-file sha256, format version, and a reservedparent_tagslot for the M2.1 diff-snapshot chain work. recipes/playwright-browser/— fork a warmed headless Chromium parent. Each child VM inherits a fully-initialised browser via mmap CoW; per-callsb.eval("await page.title()")returns in ~10–80 ms instead of the ~2 s required for a cold Chromium spawn. Requires--mem-size-mib 2048.recipes/jupyter-kernel/— SciPy-warm parent for code- interpreter workloads.forkd-agent.pyrecipe-level eval bridge. When the rootfs contains/etc/forkd-recipe.envdeclaringFORKD_WARMUP_CMDandFORKD_AGENT_LANG=node, the agent multiplexessb.eval(<js>)calls to a warmup subprocess over a line-JSON protocol.Sandbox.eval()deserialises the reply back into a native Python object.ROADMAP.mddocumenting M1 / M2 / M3 milestones.- README Snapshot Hub section, Chinese translation
(
README-zh.md), PyPI version badge.
forkd evalnow prints theresult_jsonfield returned by Node-recipe replies; previously this surface was silently dropped on the CLI side. Python recipes'result(repr-string) path unchanged for backwards compatibility.- Warmup process inside
playwright-browseremits sentinel strings (__js_Infinity__,__js_-Infinity__,__js_NaN__) for non-finite JavaScript numbers, whichJSON.stringifyotherwise silently converts tonull. Takes effect for any recipe rebuilt against 0.1.3. - Error messages on
forkd unpa 40D2 ck,forkd pull, integrity failures, manifest parse errors, and HTTP failures now show the underlyingCaused by:chain with operator-actionable hints (DNS failure, expired presigned URL, corrupted pack, etc.).
crates/forkd-cli: newhubmodule for pack format + push/pull.rootfs-init/tests/— host-runnable smoke tests for the recipe eval bridge (fake-warmup.py,smoke-test.sh,smoke-sdk.py).- CI: branch-protected
mainwithrustjob (fmt + clippy + test) required; PyPI Trusted Publisher (OIDC) workflow.
- Python SDK published to PyPI (
pip install forkd). - CubeSandbox row in the README benchmark table now leads with
the bare-metal-host context after a
systemd-detect-virt: noneproof, to address the "nested virtualisation might be skewing the numbers" concern raised upstream.
- README "Where forkd fits" rewritten with 5 concrete use cases.
- Initial GitHub Release pipeline.
- Initial public release. Fork-on-write microVM primitive, controller daemon, REST API, Python SDK, six recipes, and the N=100 spawn benchmark.