8000 perf: speed up 3 files (1.07× on the declared workload) by parthvelobyte · Pull Request #1 · parthvelobyte/llm.c · GitHub
[go: up one dir, main page]
Skip to content

perf: speed up 3 files (1.07× on the declared workload) - #1

Draft
parthvelobyte wants to merge 1 commit into
masterfrom
evo/backend-e2e-20260927
Draft

perf: speed up 3 files (1.07× on the declared workload)#1
parthvelobyte wants to merge 1 commit into
masterfrom
evo/backend-e2e-20260927

Conversation

@parthvelobyte
@parthvelobyte parthvelobyte commented Sep 28, 2026
Copy link
Copy Markdown
Owner

Summary

Changes 3 files: .evo/fleet.json b/.evo/fleet.json (+75/−0), .fleet/bench_gpt2.c b/.fleet/bench_gpt2.c (+192/−0), train_gpt2.c (+47/−24).

Workload: CPU-only synthetic GPT-2 forward/backward benchmark for train_gpt2.c reference kernels. Uses deterministic generated inputs to avoid pretrained-weight downloads and includes separate held-out entry points with different shapes.

On the declared workload the patched build runs 1.072× faster (median of 9 paired runs; CI95 1.066–1.087×) in a fresh confirmation after search.

Why it is faster

Stated by the optimizing agent; supported only by the measurements below, not independently verified.

  • Bottleneck: Optimize CPU C reference forward/backward kernels for the fixed deterministic synthetic GPT-2 workload without changing observable outputs. Preserve numerical behavior within the existing harness tolerance. Focus on reducing unnecessary work in attention_backward/matmul kernels with algebraically equivalent transformations; do not change the workload harness or evaluator.
  • Optimize CPU C reference forward/backward kernels for the fixed deterministic synthetic GPT-2 workload without changing observable outputs. Preserve numerical behavior within the existing harness tolerance. Focus on reducing unnecessary work in attention_backward/matmul kernels with algebraically equivalent transformations; do not change the workload harness or evaluator.

Evidence · tier: measured

Final confirmation passed: clean rebuilds of the pinned baseline and the patch, tests, then paired measurements with fresh (candidate-first) ordering.

Workload Role Pairs Median speedup CI95 CoV base / cand Verdict
training training 9 1.072× 1.066–1.087× 3.6% / 4.3% passed

Median = median of per-pair speedups (baseline time / candidate time; >1 is faster). CI95 = distribution-free order-statistic interval for that median (achieved coverage 96.1%). A holdout is rejected when its median < 0.99× and CI95 upper < 1.0×; "no regression" means its median was at least 0.99×, and an inconclusive holdout blocks tier tested.

No holdout: no held-out workload was declared, so generalization beyond the training workload is untested.

  • Tests: passed on clean baseline and candidate rebuilds.
  • Differential vs. baseline (exact output, untimed): passed (1/1 scenarios matched the baseline).
  • ASan + UBSan (separate untimed builds): passed (1/1 scenarios matched the baseline).
  • Noise floor ±2.62% (A/A: baseline against itself, 11 pairs; minimum detectable effect 5.57% at 7 pairs) on a bare metal; search counters: rusage (screening only, never acceptance).
    • Environment: load: 1-minute load 6.14 on 10 CPUs; other work competes for cores
  • Search-time measurement: 1.072× over 7 pairs; the table above is the independent re-run.

Build configuration

The build-configuration lane did not run, so configuration and code gains are not separated.

How to reproduce

The benchmark harness is not included. Fleet measured with files that are neither in this repository at f1e2ace65149 nor in this patch (.evo/fleet.json, .fleet/bench_gpt2.c). The commands below record what was run; they are not runnable upstream as-is.

git checkout f1e2ace651495b74ae22d45d1723443fd00ecd3a
git apply candidate_3.patch
sh -c 'mkdir -p .fleet/build && cc -Ofast -march=native -Wno-unused-result -Wno-ignored-pragmas -Wno-unknown-attributes .fleet/bench_gpt2.c -lm -o .fleet/build/bench_gpt2'
.fleet/build/bench_gpt2 check  # tests
.fleet/build/bench_gpt2 bench 2 32 64 4 2 512 512 5  # timed workload

Time the workload on the original and patched trees in alternating pairs (9 pairs in the confirmation) and compare per-pair ratios. Measured on Apple M2 Pro macos-aarch64. Held-out inputs stay sealed with the run (portfolio digest none); they are not part of this patch.

Limitations

  • Empirical evidence only; formal equivalence was not attempted.
  • The intervals do not correct for selecting this patch from 3 candidates tried; the fresh confirmation run is the guard against that bias.
  • Speedups were measured on one machine class; other hardware, compilers or inputs may differ.
  • No holdout: generalization beyond the training workload is untested.

Receipt validation_receipt.json · key sha256:766ac7fc2d69…

@parthvelobyte @claude
perf: speed up 3 files (1.07× on the declared workload)
19d7afd 
Evo Fleet patch-mode campaign run_07ba34078428496ea55b0702503c7eb4 (candidate_3).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@parthvelobyte
0