English | 简体中文
Living survey · Updated November 9, 2026 · GitHub repository
Scaling-law research repeatedly uncovers what an existing resource model leaves out, then revisits how resources should be allocated. This survey follows that development from predictable training returns and parameter–data allocation to effective data, sparse architectures, deployment costs, post-training, inference-time computation, and capability measurement.
Each research thread follows the original problem → the central insight → the mechanism → experimental evidence → remaining limitations → subsequent branches. Connections distinguish direct responses, parallel approaches, and the survey's own synthesis.
Edition status: Complete English and Chinese manuscripts and PDFs are available. This English README is the default repository homepage; a complete LaTeX build is not yet available.
English survey · English PDF · Chinese survey · Chinese PDF · References and versions · Research map (Chinese) · Coverage audit (Chinese) · BibTeX
The figure summarizes relationships between research problems; it is not an experimental plot.
Latest update: Five technical additions examine ensemble allocation, rewrite-generation costs, information-aware inference budgets, parallel agent teams, and hinted capability forecasts. The catalog contains 198 primary sources, with 184 cited in technical chapters; a METR report remains in the index and community radar. Three targeted revision checks preserve current technical conclusions. Two new dated Reddit observations bring the radar to 24 records. See the November 9 update record for evidence and access limits.
The chapter links below open the complete English edition. The Chinese homepage provides the corresponding Chinese chapters.
| Chapter | Question driving the research | Main trajectory |
|---|---|---|
| 1. Predictability and budgets | How can we predict returns before training, and why does the optimal allocation change? | Learning curves → Kaplan → Chinchilla → reanalysis, hyperparameter transfer, and training recipes |
| 2. Effective data | How can scaling continue when data are limited, unequal in value, or drawn from changing distributions? | Repetition → deduplication and selection → DoReMi / RegMix → synthesis and information retention |
| 3. Architecture and deployment | Why do parameter counts and FLOPs fail to capture practical cost on their own? | Routing and MoE → memory, I/O, and caching → long context → lifecycle optimization |
| 4. Post-training and RL | How do we turn available candidate solutions into reliable policies, and when do learning signals fail? | Self-training → feedback and verification → GRPO and successors → overoptimization, coverage, and curricula |
| 5. Inference-time computation | Should additional computation go to longer reasoning, resampling, search, or verification? | CoT and multiple samples → search and PRMs → difficulty-dependent budgets → distillation and latent depth |
| 6. Theory and capability evaluation | Where do power laws come from, and how does loss relate to usable capabilities? | Mechanisms → feature learning → emergence and measurement → extrapolation and generalization limits |
| 7. Multimodality and interaction | Which resource dimensions matter when modalities and feedback change? | Vision scaling → image–text supervision and mixtures → diffusion computation → robotics and environmental coverage |
The survey also includes an introduction, a community discussion radar, and a cross-thread synthesis. Chapters retain variable definitions, comparison protocols, experimental conditions, and open questions, while separating established foundations from preliminary findings.
The organization draws on JonnesLin/post-training-survey. We read its Chinese preface, introduction, seven chapters, conclusion, and design and implementation documents, and checked its 468 bibliography entries against 440 unique citation keys used in the Chinese chapters. We adopt its problem-driven structure, depth hierarchy, and cross-thread connections while checking technical claims against their primary sources. See the method analysis and the separate review of research organization and repository architecture, the latter in Chinese.
The initial edition contained 45 primary sources and three compressed technical chapters. The expanded edition has seven technical chapters, approximately 47,000 Chinese characters, and 198 distinct primary-source records, of which 184 are cited in the technical chapters. Catalog size, actual citation coverage, and reading depth are reported separately. Abstract-only candidates are not counted as close readings. Exact figures are available in the coverage map and machine-readable audit.
The repository records 24 traceable items from X, YouTube, and Reddit, including author announcements, research talks, engineering reproductions, and rebuttals. Engagement records retain observation dates and retrieval limitations; unavailable values remain null, and search-index snapshots are not presented as live rankings. Discussions identify questions and disagreements; papers, author experiments, and technical reports support technical judgments. See the community radar.
The main focus is language models, with theory, vision, diffusion, and robotics used to examine assumptions and boundaries. The latest incremental search extends through November 9, 2026, without a claim of exhaustive coverage. The survey does not independently reproduce the cited experiments. Individual records retain reading scope, version information, and limitations; selected 2026 results are distinguished from established findings.
A Codex maintenance task is scheduled for Mondays at 09:00 in America/Los_Angeles. It searches papers, author blogs, technical reports, and the three community platforms, then updates the narrative, source records, both language editions and PDFs, and repository after verification. Notifications are reserved for substantive changes, failures, or required user action. Execution depends on the local task environment. See UPDATING.md for the maintenance procedure, currently documented in Chinese.
- Research and writing methodology (Chinese)
- Changelog (Chinese) and publication status (Chinese)
- Source records and maintenance state
python3 scripts/build_survey.py
python3 scripts/validate.py
# PDF dependencies: reportlab, Pillow, pypdf; matplotlib for display mathematics
python3 scripts/build_pdf.py --language en --check
python3 scripts/build_pdf.py --language zh --checkThe Chinese chapters in docs/, their complete English counterparts in docs/en/, and verification records in sources/ are maintained sources. The two complete manuscripts, reference indexes, citation audit, and PDFs are generated artifacts. Translation review records detect changes in either language; an update must be reviewed before its fingerprints are renewed. Original papers, community posts, and the reference repository remain the work of their respective authors; this repository distributes its own synthesis and links, not third-party full texts.