You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Loader: llama.cpp-converted GGUFs (qwen2.* metadata keys) load as d=-1 and abort in ggml #203
A GGUF written by llama.cpp convert_hf_to_gguf.py carries general.architecture = qwen2 and qwen2.embedding_length, qwen2.block_count, ... Arch.from_gguf (lib/toy/models/arch.rb) probes llama.*, then olmoe, gemma2, qwen3moe, qwen2moe, deepseek2, and never qwen2, so every scalar reads -1:
toy describe on the same file reads the header correctly (D=3584 L=22 d_ff=9472), so the CLI and the runner disagree.
Asks:
Take the prefix from general.architecture (at least qwen2, qwen3, llama) instead of a fixed probe list.
Fail loud when any of d_model / n_layers / n_heads / d_ff is <= 0, before building a graph.
Decide and document whether llama.cpp-layout GGUFs (transposed weights, K-quants such as Q4_K_M) are a supported input. Today the only loadable route is prep/convert_smollm2_to_gguf.py, which rules out the Q4_K_M file (2.48 GB) that is the smallest shipping artifact.
Found while packaging the shrink POC student (~/sites/shrink, Qwen2.5-7B pruned to 22 layers / d_ff 9472 / 3.98 B, distilled from Qwen3.6-27B) with Toy. Toy at 6bef2ce, runners in libexec/ built 2026-08-31, gx10 (GB10, CUDA 13.0, driver 580.178.04). Test files: /srv/data/scratch/shrink/gguf/toy/t27-toy-{q8,f32}.gguf (Toy converter) and /srv/data/scratch/shrink/gguf/recovered-L22-F9472-t27-Q8_0.gguf (llama.cpp converter).
What was wrong. Two separate things stopped a llama.cpp-converted file from working:
The metadata prefix was assumed to be llama.*. A llama.cpp conversion of a Qwen2.5 model writes qwen2.*, so d_model and friends read as 0.
Weight layout was decided only by the toy.ggml_native flag. llama.cpp files are native layout but do not carry the flag, so they were read as legacy (transposed) and decoded the wrong tokens with no error.
What changed.
The shim resolves the prefix from an explicit list (llama, olmoe, gemma2, qwen3moe, qwen2moe, deepseek2, qwen2, qwen3). Anything else stops with "unsupported architecture" naming general.architecture. A zero d_model / n_layers / d_ff is now fatal rather than carried forward.
Layout is the flag if present, otherwise the shape of blk.0.ffn_gate.weight. All four places that read the flag use the same call, on CPU and CUDA.
docs/models.md has a new section, "Which GGUF files load".
Checked on the real files. The llama.cpp Q8_0 and Q4_K_M of the 3.98 B student decode the same ids as the Toy-converted Q8 (9707 11 1879 0 358 2776 264 220 16), on CPU and CUDA.
Gate.make gate-llamacpp-gguf (legs: resolve, decode, layout, foreign). All four legs fail against the pre-fix runner.
Not covered.eval_ce, eval_lmc and warm_start still read llama.embedding_length directly, so they will not take a qwen2.* file yet. The Metal runner declares the new calls but could not be built or run on this box.
Activity
OriPekelman commentedon Nov 11, 2026
Shipped on
mainat4e0be26(fix commit34e96c8).What was wrong. Two separate things stopped a llama.cpp-converted file from working:
llama.*. A llama.cpp conversion of a Qwen2.5 model writesqwen2.*, sod_modeland friends read as 0.toy.ggml_nativeflag. llama.cpp files are native layout but do not carry the flag, so they were read as legacy (transposed) and decoded the wrong tokens with no error.What changed.
llama,olmoe,gemma2,qwen3moe,qwen2moe,deepseek2,qwen2,qwen3). Anything else stops with "unsupported architecture" naminggeneral.architecture. A zerod_model/n_layers/d_ffis now fatal rather than carried forward.blk.0.ffn_gate.weight. All four places that read the flag use the same call, on CPU and CUDA.docs/models.mdhas a new section, "Which GGUF files load".Checked on the real files. The llama.cpp Q8_0 and Q4_K_M of the 3.98 B student decode the same ids as the Toy-converted Q8 (
9707 11 1879 0 358 2776 264 220 16), on CPU and CUDA.Gate.
make gate-llamacpp-gguf(legs: resolve, decode, layout, foreign). All four legs fail against the pre-fix runner.Not covered.
eval_ce,eval_lmcandwarm_startstill readllama.embedding_lengthdirectly, so they will not take aqwen2.*file yet. The Metal runner declares the new calls but could not be built or run on this box.