ACT is a research codebase for one question: can a language model's computational topology be learned during pretraining rather than fixed in advance? The repository contains the model, the training and evaluation code, the topology operations that mutate it, and the full experimental record behind every claim.
The honest summary is that the original hypothesis is not established. Random topology churn did not reliably improve real-text training, and the probe-based adaptive selector did not beat random churn. The project pivoted to a narrower question that the evidence does answer: does architecture materially affect training efficiency at these scales, and is a better architecture findable by search? See Findings and Status.
- Findings
- Status
- Installation
- Quick start
- Reproducing the results
- Repository layout
- Documentation
- Limitations
- License
| Question | Result | Where it is recorded |
|---|---|---|
| Does architecture matter at matched parameter budget? | Yes. An 8-candidate design reaches a 0.2462-nat spread against 0.0148 nats of seed noise (exact seed-matched p <= 5e-06). |
artifacts/ARCH_SEARCH_FINDINGS.md |
| Which lever? | Residual-stream width. Hidden size contributes −0.1836 nats; depth contributes +0.0783 nats, with shallow beating deep. Both reproduce on fresh seeds. | artifacts/ARCH_SEARCH_FINDINGS.md |
| Do the two interact? | Yes. The depth penalty depends on width (interaction +0.0827 nats, six-seed replication), so a search must mutate depth and width jointly. | artifacts/ARCH_SEARCH_FINDINGS.md |
| Do the findings survive longer training? | The shallow-beats-deep effect persists at 10× the training length (+0.0209 nats, exact p = 0.003086). The winner changes; the ranking holds (Spearman 0.90). |
artifacts/arch-zoo-100m-v3-crossed-horizon10x/ |
| Does active-parameter count explain loss? | No. Across a ×4.5 range under a matched budget it shows no association with loss. | artifacts/ARCH_SEARCH_FINDINGS.md |
| Can a small MoE learn real English? | Yes, as a proof of the training path. A 20.9M-parameter MoE trained on 98.3M tokens of TinyStories reaches 1.6228 nats held-out loss (perplexity 5.07), against a uniform floor of 9.0109, a unigram model at 5.9681 and a bigram at 7.2348. | artifacts/real-lm/report.json |
Every number above is recomputed from the committed ledgers by scripts/verify_record.py, not copied from prose.
- Research direction: the architecture-search phase is the active line. Further scaling is gated on the width-control design in
docs/width-control-design.md. - Gate 4 (Kaggle T4×2 campaign): execution is complete. All three queues published the declared 12-seed block, and the pre-registered verdict is
PASS_COST_ONLY_NO_DEMONSTRATED_EDGE(exit 2, Gate 4 withheld on the quality premise). The state is recorded indocs/IMPLEMENTATION_STATUS.mdandartifacts/study-summaries/gate4-verdict.json. - Code-model distillation: an experiment comparing distillation against plain cross-entropy for a small code model is scripted (
scripts/run_code_lm_campaign.sh). Its headline table is written only when the campaign completes.
┌──────────────────────────────────────────────┐
│ RESEARCH CONTROLLER │
│ evolutionary topology, experiments, cost │
├──────────────────────────────────────────────┤
│ TOPOLOGY SYSTEM │
│ grow, prune, merge, split, rewire │
├──────────────────────────────────────────────┤
│ MODEL CORE │
│ attention, gated delta net, MoE, residual │
├──────────────────────────────────────────────┤
│ TRAINING ENGINE │
│ PyTorch, CUDA 12.6 (T4x2 on Kaggle) │
└──────────────────────────────────────────────┘
- Topology mutation.
prune_expert,grow_expert,clone_expert,merge_expertsandsplit_expertact on MoE experts;rewire_layerchanges a layer's residual input edge;change_moduleswaps a layer's sequence mixer or feed-forward type;toggle_layerchanges effective depth. Uniform active-expert invariants are enforced at build time. - Importance metrics. Routing load, parameter and gradient norms, Fisher approximations (
fisher_ema), functional redundancy, temporal stability, and causal ablation (expert_ablation_delta). - Prior-ranked module search. A
SearchPriorranks a genome's layer and module candidates deterministically throughrank_module_candidates, with a rationale for each candidate. No external model artifacts are bundled. - Named mixer layouts. The
gdn_3to1schedule (three gated-delta-net layers to one attention layer) among others, with a committed throughput benchmark inartifacts/mixer-benchmark/. - Deterministic token shards. Checksum-addressed synthetic shards with bounded reads (
build_synthetic_shards,read_tokens), the contract real binary shards will follow. - Kaggle-ready execution. Stateless resume, dataset sharding with checksums, and safety budgets for T4x2 sessions.
Requires Python 3.10 or newer. The project pins the CUDA 12.6 PyTorch wheel so Pascal GPUs such as the GTX 1070 (compute capability 6.1) receive sm_61 kernels.
uv sync --extra devUse --device cpu on any command to force CPU execution, which is what the deterministic studies use.
Train a small topology model for a few steps:
PYTHONPATH=src uv run python -m act.training.cli --steps 8Run the Experiment 0 topology gate, which compares fixed, random-churn and adaptive arms under identical batches and mutation budgets:
PYTHONPATH=src python -m act.evolution.cli --seeds 1,2,3,42 --steps 48Train and sample the small language model on TinyStories:
PYTHONPATH=src python scripts/train_real_lm.py --max-steps 12000
PYTHONPATH=src python scripts/real_lm_infer.py --demoThe checkpoint is a run output and is not committed, so real_lm_infer.py needs the training command to have run first.
Run the test suite:
PYTHONPATH=src python -m pytest -qThe suite runs in parallel by default (pytest-xdist); pass -n 0 for a serial run.
Check that every documented claim still recomputes from its ledger, and that each committed record regenerates from its own documented command:
PYTHONPATH=src python scripts/verify_record.py
PYTHONPATH=src python scripts/check_record_freshness.py --strict-rankTo check the same things against a clean clone of HEAD (the view a reader of the repository gets), run:
PYTHONPATH=src python scripts/check_fresh_clone.pyAnalysis of a sweep is run with scripts/arch_zoo_analyze.py; the screening and confirmation commands for each arm are in that arm's NOTES.md.
src/act/ the library
config/ configuration schemas
data/ deterministic token shards and readers
evaluation/ metrics and held-out evaluation
evolution/ the topology gate and its CLI
importance/ importance metrics (routing load, Fisher, ablation)
model/ the MoE causal language model
ops/ Kaggle remote execution and handoff publishing
search/ module ranking and architecture-search reporting
topology/ mutation operations (prune, grow, clone, merge, split, rewire)
training/ training engine, session runner and CLI
scripts/ research instruments and campaign drivers
tests/ the test suite
notebooks/ generated notebooks (rebuilt by scripts/build_notebooks.py)
docs/ plan, research report, status log, runbooks and design notes
artifacts/ committed experiment ledgers, records and analyses
The standing documents are under docs/. Each has one job:
docs/complete_implementation_plan.mdis the plan of record: the research questions, hardware strategy, model and search-space design, and gates. It is prescriptive.docs/RESEARCH_REPORT.mdis the narrative report of results, with each study's pre-registration beside its outcome.docs/IMPLEMENTATION_STATUS.mdis the implementation log. It is append-only and newest-first.docs/T4_RUNBOOK.mdis the operational runbook for Kaggle sessions.docs/handoff-sharding-design.mddescribes how checkpoint handoffs are split across Kaggle dataset versions.docs/width-control-design.mdis a design note for the next gate: width mutation and its pre-registered outcomes.docs/training-throughput.mdis a measurement note on where a training run's wall-clock time goes.docs/CONTRIBUTING.mddescribes setup, the checks a change must pass, and the record conventions.
Per-arm material lives beside each experiment under artifacts/:
artifacts/ARCH_SEARCH_FINDINGS.mdsynthesises the architecture-search evidence.artifacts/*/NOTES.mdholds each arm's pre-registration and the command that reproduces its record.artifacts/*/decision.mdis generated from the ledgers; regenerate it rather than editing it.artifacts/*/analysis.md,artifacts/*/FINDINGS.md,artifacts/*/PERFORMANCE_NOTES.mdandartifacts/*/SCREENING_AUDIT.mdare per-arm and per-benchmark notes. They are not machine-checked.
The index above is itself checked by tests/test_doc_index.py.
Where to start. Read docs/RESEARCH_REPORT.md for the findings, then the newest entries of docs/IMPLEMENTATION_STATUS.md for the current state. The plan in docs/complete_implementation_plan.md is the reference for what was intended, not what was built.
- Scale. The largest models here are about 100M parameters, trained for a fraction of the token budget of comparable published models. The findings are about relative architecture effects at these scales, not absolute capability.
- Hardware. The local development card is a GTX 1070 (fp32 in practice). The Kaggle T4 path runs fp16. Comparisons whose effect size is near the measured fp16 difference are kept in fp32.
- Data. The language-model proof uses TinyStories, a children's corpus. It shows that the training path works, not that the architecture hypothesis holds.
- Negative results are recorded, not hidden. The original random-churn and adaptive-selector hypotheses did not hold. Several effects stay open or are reported as not established.
Copyright (c) 2026 James Summerill. All rights reserved. No part of this repository may be used, copied, modified, distributed or sold without prior written permission from the copyright holder. See LICENSE.