8000 Loader: llama.cpp-converted GGUFs (qwen2.* metadata keys) load as d=-1 and abort in ggml · Issue #203 · OriPekelman/toy · GitHub
[go: up one dir, main page]

Skip to content

Loader: llama.cpp-converted GGUFs (qwen2.* metadata keys) load as d=-1 and abort in ggml #203

Open

Description

@OriPekelman

A GGUF written by llama.cpp convert_hf_to_gguf.py carries general.architecture = qwen2 and qwen2.embedding_length, qwen2.block_count, ... Arch.from_gguf (lib/toy/models/arch.rb) probes llama.*, then olmoe, gemma2, qwen3moe, qwen2moe, deepseek2, and never qwen2, so every scalar reads -1:

Arch(qwen2, vocab=152064, d=-1, L=-1, n_q=-1, n_kv=-1, d_ff=-1, qkv_bias=true, rope_base=0.0, rms eps=0.0, swiglu)
loading ... → FFI direct (267 tensors)
ggml-backend.cpp:327: GGML_ASSERT(buf != NULL && "tensor buffer not set") failed

toy describe on the same file reads the header correctly (D=3584 L=22 d_ff=9472), so the CLI and the runner disagree.

Asks:

  1. Take the prefix from general.architecture (at least qwen2, qwen3, llama) instead of a fixed probe list.
  2. Fail loud when any of d_model / n_layers / n_heads / d_ff is <= 0, before building a graph.
  3. Decide and document whether llama.cpp-layout GGUFs (transposed weights, K-quants such as Q4_K_M) are a supported input. Today the only loadable route is prep/convert_smollm2_to_gguf.py, which rules out the Q4_K_M file (2.48 GB) that is the smallest shipping artifact.

Found while packaging the shrink POC student (~/sites/shrink, Qwen2.5-7B pruned to 22 layers / d_ff 9472 / 3.98 B, distilled from Qwen3.6-27B) with Toy. Toy at 6bef2ce, runners in libexec/ built 2026-08-31, gx10 (GB10, CUDA 13.0, driver 580.178.04). Test files: /srv/data/scratch/shrink/gguf/toy/t27-toy-{q8,f32}.gguf (Toy converter) and /srv/data/scratch/shrink/gguf/recovered-L22-F9472-t27-Q8_0.gguf (llama.cpp converter).

Activity

  1. OriPekelman commented on Nov 11, 2026

    @OriPekelman
    OwnerAuthor

    Shipped on main at 4e0be26 (fix commit 34e96c8).

    What was wrong. Two separate things stopped a llama.cpp-converted file from working:

    1. The metadata prefix was assumed to be llama.*. A llama.cpp conversion of a Qwen2.5 model writes qwen2.*, so d_model and friends read as 0.
    2. Weight layout was decided only by the toy.ggml_native flag. llama.cpp files are native layout but do not carry the flag, so they were read as legacy (transposed) and decoded the wrong tokens with no error.

    What changed.

    • The shim resolves the prefix from an explicit list (llama, olmoe, gemma2, qwen3moe, qwen2moe, deepseek2, qwen2, qwen3). Anything else stops with "unsupported architecture" naming general.architecture. A zero d_model / n_layers / d_ff is now fatal rather than carried forward.
    • Layout is the flag if present, otherwise the shape of blk.0.ffn_gate.weight. All four places that read the flag use the same call, on CPU and CUDA.
    • docs/models.md has a new section, "Which GGUF files load".

    Checked on the real files. The llama.cpp Q8_0 and Q4_K_M of the 3.98 B student decode the same ids as the Toy-converted Q8 (9707 11 1879 0 358 2776 264 220 16), on CPU and CUDA.

    Gate. make gate-llamacpp-gguf (legs: resolve, decode, layout, foreign). All four legs fail against the pre-fix runner.

    Not covered. eval_ce, eval_lmc and warm_start still read llama.embedding_length directly, so they will not take a qwen2.* file yet. The Metal runner declares the new calls but could not be built or run on this box.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0