8000 GitHub - erogol/BlaGPT: Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible experimentation and exploration. · GitHub
[go: up one dir, main page]

Skip to content
erogolPublic

About

Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible experimentation and exploration.

Topics

Resources

Stars

118 stars

Watchers

8 watching

Forks

Repository files navigation

BlaGPT

A playground to test language model architectures, layers and training tricks on a small budget. Each new idea goes on top of the best config. It stays only if the final val loss drops.

👑 Current best

Item Value
Val loss 3.1153 (5100 steps, 8x H200)
Model 10 layers, d=768, 404.64M params (193M of them in the Engram table)
Main parts Latent Lattice 8000 · RoPE + RoVE · Aurora optimizer · Engram (vocab ×5)
Speed 732 ms/step, peak 70,569 MiB per GPU
Config best_model_config.py · ar/best_config.json
torchrun --standalone --nproc_per_node=8 bla_gpt/train.py --run_name best_model --model_name best

Best val loss after each kept change

Contents: Progression · Alternative architectures · Byte-level models · Optimizers · Adding a new model · Training · Techniques

Progression

Each row adds one change to the best model at that time. ✅ = kept, ❌ = discarded, 👑 = current best. — = not recorded in the original run.

Technique Val loss Δ Peak MiB ms/step Notes
Setting base theta to 1_000_000
llama3
— 3.3324 — — — increased the final validation loss
Z-loss regularization
link
— 3.3527 — — — increased the final validation loss by 0.02
KV-Shifting attention
link
— 3.3138 -0.0172 42858 — lower loss than its base (3.3310)
Dilated Attention (LongNet)
link
— — — — — —
Multi-Head Latent Attention
link
— 3.3479 — 42192 — —
Per token output bias — 3.3257 — 42120 — —
DyT Norm
link
— — — — — loss stayed high; did not train well
Forgetting Transformer (Vanilla and Pro vers)
link
— 3.3243 — — — vanilla version; the Pro version ran out of memory
Multi-Token Attention
link
— 3.3357 — 42136 — —
Differential Attention
link
❌ 3.2460 +0.0049 41521 — —
Softpick
link
— 3.3446 — 59417 — —
Canon Layer
link
— 3.3217 — 43199 — —
Parallel Transformer Block
link
— 3.3473 — 40302 — —
Per Layer Token Embedding
link
— 3.2411 — 40916 — —
PolyNorm
link
❌ 3.3017 +0.0606 40895 — —
PolyReLU
link
❌ 3.2642 +0.0231 40890 — —
TOP loss
paper · explanation
❌ 3.2636 +0.0225 47816 — —
Simplified RoPe
link
❌ 3.2620 +0.0209 43585 388.54 —
Gated Attention
paper · explanation
✅ 3.2327 -0.0084 45968 413.01 —
ResFormer (Plus)
paper · explanation
❌ 3.3538 +0.1211 38223 326.32 —
Engram (My Simple Variant)
paper · explanation
✅ 3.2296 -0.0031 50488 504.09 —
Differential Attention v2
paper · explanation
✅ 3.2274 -0.0022 52829 535.16 —
Exclusive Self Attention
paper
✅ 3.2303 -0.0024 49859 424.54 tested on the Gated Attention base (3.2327), in parallel with the two rows above
Hyper-Connections
paper · explanation
❌ 3.2741 +0.0414 68729 928.26 tested on the Gated Attention base (3.2327)
All keeps combined (5100 steps) ✅ 3.2354 — — — new 5100-step base: all kept techniques above in one config (ar/full_train/combined_keeps.json); every row below uses this 5100-step setup
GOAT Sink Prior
paper · explanation
✅ 3.2298 -0.0056 52231 377.61 —
GatedNorm
paper · explanation
✅ 3.2230 -0.0068 50023 410.68 —
HybridNorm
paper · explanation
✅ 3.2224 -0.0006 50495 474.82 —
Composable Gated Attention
paper · explanation
✅ 3.2128 -0.0096 58553 489.07 —
Composable Value Residual
paper · explanation
✅ 3.2011 -0.0117 58988 497.55 —
U-net Long Skips
source · explanation
✅ 3.1979 -0.0032 58632 494.31 —
Learning rate 0.001 -> 0.0014
Muon source
✅ 3.1965 -0.0014 58632 491.00 —
GRAPE-A Positional Encoding
paper · explanation
✅ 3.1897 -0.0068 59643 580.32 replaces RoPE; later replaced again by RoPE + RoVE
Aurora Optimizer
paper · explanation
✅ 3.1603 -0.0294 59643 721.25 replaces Muon, learning_rate=0.03
Weight EMA + LR floor ✅ 3.1487 -0.0116 61239 731.65 LR decays to 15% of peak, not to 0, and eval uses the average weights of the last 300 steps; removed in F107b_e5
Token normalization for Engram
PR #375 · explanation
❌ 3.1564 -0.0039 59643 719.88 tested on the Aurora base (3.1603), in parallel with Weight EMA + LR floor; tokens with the same first 2 bytes share one Engram entry; off by default
RoPE + RoVE + Latent Lattice (F107b_e5)
RoPE · RoVE · Latent Lattice · explanation
👑 3.1153 -0.0334 70569 732.44 404.64M params; RoPE + RoVE replace GRAPE-A; one recurrent latent (width 256) on layers 2-7 modulates q/k/v and the MLP (FiLM after the QK norm, Lattice params in AdamW lr 1e-3, vstream); weight EMA and LR floor removed; second seed: 3.1230

Early techniques with no recorded loss: Multi-token prediction · Weight tying · Grouped query attention · Capping logits · QKV bias · Zero-init projection layer · Post and pre-RMSNorm.

Run logs: ar/full_results.tsv (full runs), ar/results.tsv (short runs), ar/best_config.json (best config). Technique notes: techniques/.

Alternative Architectures

Model Val loss Peak MiB ms/step Notes
MegaByte
link
3.810 — — —
FTP (heavily modified)
link
3.901 — — —
Rene
link
3.340 — — —
Rwkv7
link
4.450 — — —
Zamba2
link
— — — no loss recorded; ranked Zamba2 > Rene > Rwkv7
Hourglass Transformer (modified)
link
3.710 — — token-level run; ranked Hourglass > MegaByte > FTP
Hymba
link
4.7505 — — much slower per step than the Transformer
Tokenformer (in BlaGPT model)
link
3.390 — — —
LLaDa (dLLM)
link
8.6930 — — diffusion loss; cross-entropy 4.2891 compares with the other models (from bla_gpt/llada_validation_cross_entropy.py)
Avey
link
3.323 51962 2871 batch size 8; very slow to train; more than 3x the memory of other models
LFM2
link
— — — TBD
Kimi Delta Attention (1:3 interleaved Full Attention)
link
3.2411→3.2532 47391 568.1 —

Byte-Level Models

Val loss here is per byte, not per BPE token. Do not compare it with the tables above.

Model Val loss Peak MiB ms/step Notes
Hourglass Transformer (modified)
link
1.0048 — 524.76 byte-level run of the token-level model above
AUNet
link
1.1502 — 1423.60 —
SpaceByte
link
1.6755 27781 423.36 —
HNet
link
1.4554 23948 433.75 —

Optimizers

Optimizer Val loss Peak MiB ms/step Notes
PaLMForeachSOAP
link
— — — best loss of the early optimizers, but almost 2x slower than Adam
Ademamix
link
— — — unstable at all tested learning rates
Adopt
link
— — — loss became NaN
CAdamW
link
3.3517 — — —
AdamW with independent weight decay
link
3.320 — — —
Adam 3.3224 — — —
AdamW 3.3310 42053 533 —
DeMo
link
3.4676 41534 820 saves 7 GB per GPU; higher loss and slower steps than Adam
Adam-Mini 3.3324 41534 610 higher loss and slower steps than Adam and AdamW; saves a little memory
MARS
link
3.3459 40953 628 —
Muon
link
3.2923 40332 620.24 —
AdaMuon
paper · explanation
— — — Adaptive Muon with second-moment estimation
Aurora
paper · tilde-research/aurora-release
3.1897→3.1603 59643 721 matrix weights use Aurora, other weights use AdamW; lr=0.03, weight_decay=0.025
BiClip
link
7.2292 39751 510 did not train well
NorMuon
paper · explanation
3.2411→3.4630 44154 387.46 —
Cautious Weight Decay
paper · explanation
3.2327→3.2334 45971 434.2 —

Adding a New Model

  • Implement the model. The forward function must return the loss.
  • Add a function register_<name> with the @register_model decorator in bla_gpt/model_registers.py.
  • Start training with --model_name <name>.

See the existing functions in model_registers.py for examples.

Training

Training scripts, quick start, datasets, tokenizers, LR finder

BlaGPT provides two training scripts:

Script Purpose Data Sources
train.py Quick experimentation with pre-tokenized data Binary shards only
train_flex.py Flexible training with custom datasets/tokenizers Binary shards, HF streaming, byte-level

Quick Start (train.py)

For rapid architecture benchmarking with the default FineWeb10B dataset:

# Get the pre-tokenized data
python data/fineweb10B_cached.py

# Start training
torchrun --standalone --nproc_per_node=8 bla_gpt/train.py --run_name my_experiment --model_name blagpt

Flexible Training (train_flex.py)

For custom datasets, different tokenizers, or HuggingFace streaming:

Using Pre-tokenized Binary Shards (Default)

torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
    --model_name blagpt \
    --run_name my_experiment

Using HuggingFace Streaming Datasets

Stream data directly from HuggingFace Hub - no pre-download required:

# FineWeb with GPT-2 tokenizer (tiktoken)
torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
    --model_name blagpt \
    --run_name hf_streaming_test \
    --use_hf_streaming \
    --hf_dataset "HuggingFaceFW/fineweb" \
    --hf_dataset_config "sample-10BT"

# C4 dataset with HuggingFace tokenizer
torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
    --model_name blagpt \
    --run_name c4_experiment \
    --use_hf_streaming \
    --hf_dataset "allenai/c4" \
    --hf_dataset_config "en" \
    --tokenizer_backend huggingface \
    --tokenizer_name "gpt2"

# Custom dataset with custom text column
torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
    --model_name blagpt \
    --run_name custom_data \
    --use_hf_streaming \
    --hf_dataset "username/my-dataset" \
    --hf_text_column "content"

Dataset Options

Option Default Description
--use_hf_streaming False Enable HuggingFace streaming mode
--hf_dataset HuggingFaceFW/fineweb HuggingFace dataset name or path
--hf_dataset_config sample-10BT Dataset configuration/subset
--hf_split train Training split name
--hf_text_column text Column containing text data
--hf_val_dataset Same as train Separate validation dataset (optional)
--hf_val_dataset_config Same as train Validation dataset config
--hf_val_split train Validation split name
--hf_val_samples 1000 Number of samples for validation (first N of val split)
--hf_shuffle_buffer 10000 Shuffle buffer size for streaming

Train/Validation Split: By default, the first 1000 samples are used for validation and training skips these samples (no overlap). When using a separate validation dataset or split, this behavior is automatically disabled.

Tokenizer Options

Option Default Description
--tokenizer_backend tiktoken Backend: tiktoken or huggingface
--tokenizer_name gpt2 Tokenizer name (encoding or model path)

tiktoken encodings: gpt2, cl100k_base, o200k_base

HuggingFace tokenizers: Any model name, e.g., meta-llama/Llama-2-7b-hf, mistralai/Mistral-7B-v0.1

Learning Rate Finder (Optional)

Run before training to find optimal learning rate:

torchrun --standalone --nproc_per_node=8 bla_gpt/find_lr.py --model_name blagpt

# Output
Results:
Steepest gradient learning rate: 3.31e-06
Elbow point learning rate: 1.20e-01
Plot saved to: logs/lr_finder_blagpt/lr_finder_plot.png
Results saved to: logs/lr_finder_blagpt/lr_finder_results.pt

Acknowledgements

The initial code is based on

Nano GPT - link

Modded NanoGPT - link

Thanks to @xumingyu2021 for memory friendly implementation of the Differential Attention

About

Experimental playground for benchmarking language model (LM) architectures, layers, and tricks on smaller datasets. Designed for flexible experimentation and exploration.

Topics

Resources

Stars

118 stars

Watchers

8 watching

Forks

Releases

Packages

Used by

Contributors

Languages

0