An extremely simple implementation of the 27b version of Gemma 3.
All important files are in core, all other code is boring and maintained by AI.
gemma_forward_parralel.py. Defines a very simple forward function, so far very optimized for prefill and not at all great for autoregressive sampling (no KV cache, no flash attention).supervised_train.pySimple next token prediction training utilities using the forward pass.rl.pyGRPO training logic for radicating integers (WIP).gemma_forward_inference.pyInference forward logic (forward_single, KV cache prefill).
entrypoints/rl.pyRL training script.entrypoints/supervised.pySupervised training script.entrypoints/gemma_forward_inference.pyInference runnable/test script (main).entrypoints/gemma_forward.pyForward-pass runnable/test script (main).
This is very much a work in progress, but the current code already samples not obviously stupid responses.
- Flash attention1
- All other inference optimizations I can come up with
- Commit the HLO dumps so we can easily spot bisect expected XLA performance issues relative to code changes
- RL memory ownership at the train-loop boundary is unclear.
mainis not jitted (logging/checkpoint side effects), so JAX cannot infer host-side liveness across iterations. Missing donation may keep old and new parameter buffers alive simultaneously, which could result in 2x parameter-buffer footprint. (Similar to the KV cache problem we had earlier).
Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC).
Footnotes
-
The reason I haven't done this yet is because I want to implement it myself and JAX primitives are insufficient, so I will have to go one level lower and implement it in Pallas. But first I have to learn Pallas, so this might take a week or two. ↩