You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
toy pack: one self-contained executable with the model, tokenizer, KV prefix and server embedded #211
End goal of the shrink POC: ship a workload-specialized model as one dependency-free file. Today the pieces are a runner binary (libexec/toy-serve, 2.8 MB CPU; the CUDA runners are 713 MB because they link the CUDA archive statically), a GGUF next to it, and env vars.
Proposal: toy pack <model.gguf> --prefix system.txt --name NAME --device cpu|cuda -o artifact producing an executable that carries the GGUF as an appended or linked section (mmap from /proc/self/exe at an offset keeps zero-copy loading), the precomputed KV prefix, and a fixed config (context, model name, port default), and that starts the OpenAI server with no arguments. artifact --describe prints the algorithm card and the provenance (source model hash, toy/spinel/ggml commits).
Open questions: whether the GGUF loader can take an (fd, offset, length) triple instead of a path; how the CUDA build links (static CUDA archive size, sm_121 kernels); whether Spinel can emit a binary with a payload section, or the payload is appended after link.
Depends on the chat-serving and KV-prefix issues.
Found while packaging the shrink POC student (~/sites/shrink, Qwen2.5-7B pruned to 22 layers / d_ff 9472 / 3.98 B, distilled from Qwen3.6-27B) with Toy. Toy at 6bef2ce, runners in libexec/ built 2026-08-31, gx10 (GB10, CUDA 13.0, driver 580.178.04). Test files: /srv/data/scratch/shrink/gguf/toy/t27-toy-{q8,f32}.gguf (Toy converter) and /srv/data/scratch/shrink/gguf/recovered-L22-F9472-t27-Q8_0.gguf (llama.cpp converter).
toy pack student.gguf --prefix system.txt --name shrink-t27 --device cuda -o shrink-t27
./shrink-t27 # serves on the packed port
./shrink-t27 --port 8080 # or PORT=8080
./shrink-t27 --describe # card and provenance; loads nothing
Result on the student./srv/data/scratch/toy-211/shrink-t27 is one 4.67 GiB file: the 3.94 GiB Q8 GGUF, the CUDA serve runner (0.67 GiB), the KV state of a 753-token system turn (68 MB), a config and a description. Started with env -i from another directory and no arguments, it:
is ready in about 8 s, restoring the 753-token prefix from itself in 41 ms;
answers "What is the capital of France?" with "Paris";
Packing took 1 min 42 s, most of it hashing and copying the model.
The open questions.
(fd, offset, length) loader: yes. ggml already has gguf_init_from_file_ptr; the shim now parses the GGUF from the executable at the entry's offset and maps that region. Weights are read in place, as from a file; nothing is unpacked at start. The gate checks the mapping in /proc/<pid>/maps.
Payload section or append: appended after link. Each file starts on a 64 KiB boundary, followed by a text index and a 24-byte trailer. Nothing is needed from Spinel.
CUDA linking: unchanged. The CUDA runtime is linked statically (hence 0.67 GiB of runner); the artifact still needs the driver's libcuda.so.1.
What a pack fixes. Model, name, context, stop ids and KV prefix come from the artifact, and MODEL_PATH, MODEL_NAME, MAX_T, STOP_IDS, PREFIX_FILE and PREFIX_CACHE in the environment are ignored. Only the port can be moved (--port, then PORT, then the packed default).
--describe prints the toy describe card, the model's sha256, the toy, spinel and ggml commits, the runner's hash, the prefix's hash and token count, the config, and the shared libraries the artifact needs.
Not dependency-free yet. The artifact is one file but it is dynamically linked:
CPU: libc, libstdc++, libm, libgcc, and through tep OpenSSL (libssl, libcrypto) and libsqlite3.
CUDA: the same plus libcuda.so.1.
A static CPU build would need static OpenSSL and SQLite, or a tep build without them. I have not tried either.
Other limits.
Tied to the OS, CPU architecture and (for CUDA) GPU class it was packed on.
No checksum over the payload. A cut file or damaged trailer refuses to start, and so does a KV prefix from another model, but a flipped byte inside the weights is not detected.
strip or code signing would remove the payload. macOS has the code path (_NSGetExecutablePath) but is not built or run.
The KV prefix matches one system message exactly. Other requests are answered, with their whole prompt evaluated.
Everything open on the serve side still applies to a pack: greedy only, one cached sequence, tep#276.
Gate.make gate-pack and gate-pack-cuda, on SmolLM2-135M:
with an empty environment, no arguments and an empty directory the artifact serves the packed name on the packed port;
its replies equal the plain runner's on the GGUF file;
the KV prefix is restored from the executable, not evaluated;
the model is mapped read-only from the executable at the entry's offset;
environment settings do not change what it serves (this leg fails when I let the environment win);
--describe carries the sha256 and commits and loads nothing;
a cut file, a damaged trailer and a foreign KV entry each refuse to start.
Full battery: ALL GATES PASS (72 legs).
One loose end: the first CUDA run of the gate timed out waiting for a server in the fixed leg. It did not recur in 8 further runs or the battery, and I had not kept that run's logs; the gate now prints the server's last log lines if it happens again.
The intermittent gate-pack-cuda timeout ("server never became ready") reported above as unexplained is found and fixed on main at 20049c6.
Cause. tep 0.11.x writes 54 bytes of test frames to file descriptor 0 at every server start (its boot-time type seeds). On a pipe, file or terminal the writes fail harmlessly. When stdin is a socket that nobody reads, they pile up, and once it fills the next start blocks before the server prints its first line. The battery's CUDA legs pass their caller's stdin to the servers they start, and under Claude Code that is a long-lived socket. With a fresh unread socket as stdin, the 20th start of a packed artifact hangs.
Fix.prep/patch_vendored_tep.rb (run by make vendor-tep) now moves those seeds to descriptor -1. With it, 300 starts on an unread socket never hang and write 0 bytes. vendor/ is not in git, so other checkouts need make vendor-tep rerun.
Gate.gate-pack and gate-pack-cuda have a new stdin leg: a start with a socket on stdin must write nothing to it. The pre-fix artifact fails it with 54 bytes. Full battery: ALL GATES PASS (73 legs).
Upstream: OriPekelman/tep#283 (0.11.x backport ask; tep main no longer has that startup code).
Not explained: an earlier 30-run loop of the gate passed in the same harness. I assume that command's stdin socket was a different or emptier one; I did not verify it.
Activity
OriPekelman commentedon Nov 13, 2026
Shipped on
mainat6b3ef9e.Result on the student.
/srv/data/scratch/toy-211/shrink-t27is one 4.67 GiB file: the 3.94 GiB Q8 GGUF, the CUDA serve runner (0.67 GiB), the KV state of a 753-token system turn (68 MB), a config and a description. Started withenv -ifrom another directory and no arguments, it:toy serve --prefix-cacheon the separate files.Packing took 1 min 42 s, most of it hashing and copying the model.
The open questions.
gguf_init_from_file_ptr; the shim now parses the GGUF from the executable at the entry's offset and maps that region. Weights are read in place, as from a file; nothing is unpacked at start. The gate checks the mapping in/proc/<pid>/maps.libcuda.so.1.What a pack fixes. Model, name, context, stop ids and KV prefix come from the artifact, and
MODEL_PATH,MODEL_NAME,MAX_T,STOP_IDS,PREFIX_FILEandPREFIX_CACHEin the environment are ignored. Only the port can be moved (--port, thenPORT, then the packed default).--describeprints thetoy describecard, the model's sha256, the toy, spinel and ggml commits, the runner's hash, the prefix's hash and token count, the config, and the shared libraries the artifact needs.Not dependency-free yet. The artifact is one file but it is dynamically linked:
libssl,libcrypto) andlibsqlite3.libcuda.so.1.A static CPU build would need static OpenSSL and SQLite, or a tep build without them. I have not tried either.
Other limits.
stripor code signing would remove the payload. macOS has the code path (_NSGetExecutablePath) but is not built or run.Gate.
make gate-packandgate-pack-cuda, on SmolLM2-135M:--describecarries the sha256 and commits and loads nothing;Full battery: ALL GATES PASS (72 legs).
One loose end: the first CUDA run of the gate timed out waiting for a server in the
fixedleg. It did not recur in 8 further runs or the battery, and I had not kept that run's logs; the gate now prints the server's last log lines if it happens again.OriPekelman commentedon Nov 15, 2026
The intermittent
gate-pack-cudatimeout ("server never became ready") reported above as unexplained is found and fixed onmainat20049c6.Cause. tep 0.11.x writes 54 bytes of test frames to file descriptor 0 at every server start (its boot-time type seeds). On a pipe, file or terminal the writes fail harmlessly. When stdin is a socket that nobody reads, they pile up, and once it fills the next start blocks before the server prints its first line. The battery's CUDA legs pass their caller's stdin to the servers they start, and under Claude Code that is a long-lived socket. With a fresh unread socket as stdin, the 20th start of a packed artifact hangs.
Fix.
prep/patch_vendored_tep.rb(run bymake vendor-tep) now moves those seeds to descriptor -1. With it, 300 starts on an unread socket never hang and write 0 bytes.vendor/is not in git, so other checkouts needmake vendor-teprerun.Gate.
gate-packandgate-pack-cudahave a newstdinleg: a start with a socket on stdin must write nothing to it. The pre-fix artifact fails it with 54 bytes. Full battery: ALL GATES PASS (73 legs).Upstream: OriPekelman/tep#283 (0.11.x backport ask; tep
mainno longer has that startup code).Not explained: an earlier 30-run loop of the gate passed in the same harness. I assume that command's stdin socket was a different or emptier one; I did not verify it.