[go: up one dir, main page]
Sign in Subscribe

What I'm Reading

How I Read Papers

I read the abstract and conclusion first to understand what's being claimed and whether it held up. Then the introduction for context and motivation. Then the experiments — what did they measure, against what baseline, and does the result actually support the claim. Methods last, once I know what I'm reading them for.

If a paper feels dense, I find one good explainer (Lilian Weng's blog, Distill.pub, or a good YouTube walkthrough), read that, then go back to the original. The goal is always to have read the original.

I keep a one-paragraph note after each paper — not a summary, but my reaction. What surprised me. What I now understand differently. That note is usually worth more than anything I underlined.

The List

Foundations & Architecture

  1. A Fast Learning Algorithm for Deep Belief Nets — Hinton, Osindero & Teh (2006). The paper that restarted deep learning after the AI winter. Layer-by-layer pre-training using RBMs proved deep architectures could work.
  2. ImageNet Classification with Deep CNNs (AlexNet) — Krizhevsky, Sutskever & Hinton (2012). The result that ended the debate. Dropped ImageNet top-5 error from 26% to 15% and made GPU-trained deep nets the default.
  3. Dropout: A Simple Way to Prevent Neural Networks from Overfitting — Srivastava et al. (2014). Randomly deactivating neurons during training forces redundant representations and dramatically reduces overfitting. Now standard in almost every architecture.
  4. Batch Normalization: Accelerating Deep Network Training — Ioffe & Szegedy (2015). Normalising activations within each mini-batch stabilised training, allowed higher learning rates, and made very deep architectures trainable.
  5. Deep Residual Learning for Image Recognition (ResNet) — He, Zhang, Ren & Sun (2015). Skip connections let gradients bypass layers during early training. Enabled networks of 100+ layers and won ILSVRC 2015. Residual connections are now everywhere.
  6. Adam: A Method for Stochastic Optimization — Kingma & Ba (2015). Combines momentum and per-parameter adaptive learning rates. The default optimiser for training neural networks, used almost universally without much tuning.

Transformers & Language Models

  1. Attention Is All You Need — Vaswani, Shazeer, Parmar et al. (2017). Replaced recurrence with self-attention entirely. Every language model, vision transformer, and multimodal model today is an implementation of this architecture.
  2. BERT: Pre-training of Deep Bidirectional Transformers — Devlin, Chang, Lee & Toutanova (2018). Masked language modelling on unlabelled text produces representations that fine-tune to nearly any NLP task. Established the pre-train/fine-tune paradigm for all of AI.
  3. Language Models are Few-Shot Learners (GPT-3) — Brown et al. (2020). 175B parameters, no task-specific fine-tuning. Proved that scale alone produces emergent capabilities — chain-of-thought, code generation, arithmetic — that smaller models simply don't have.
  4. Training Language Models to Follow Instructions (InstructGPT) — Ouyang et al. (2022). Introduced RLHF for language models: fine-tune on demonstrations, train a reward model on human preferences, optimise with RL. Made GPT into something people actually wanted to use. ChatGPT follows directly from this.
  5. LLaMA: Open and Efficient Foundation Language Models — Touvron et al. (2023). Showed that smaller models trained on more tokens outperform larger models trained on less data. Democratised foundation model research and sparked the open-source LLM ecosystem.
  6. Llama 2: Open Foundation and Fine-Tuned Chat Models — Touvron et al. (2023). Extended LLaMA with RLHF-tuned chat variants and detailed safety methodology. One of the most studied open models in the field.
  7. Llama 3 Herd of Models — Meta (2024). Detailed the full training pipeline for a frontier-class open model at scale, including data curation, post-training, and multimodal extension.

Generative Models

  1. Auto-Encoding Variational Bayes (VAE) — Kingma & Welling (2013). Learned structured latent spaces using the reparameterisation trick for end-to-end training. The probabilistic framework here is foundational to modern diffusion models.
  2. Generative Adversarial Nets (GANs) — Goodfellow et al. (2014). Adversarial training between generator and discriminator produced the first sharp, realistic generative outputs. Dominated image generation for the late 2010s.
  3. Denoising Diffusion Probabilistic Models (DDPM) — Ho, Jain & Abbeel (2020). Learn to reverse a gradual noising process. Produced images of quality and diversity GANs couldn't match. Behind DALL-E, Stable Diffusion, Midjourney, and Sora.
  4. High-Resolution Image Synthesis with Latent Diffusion Models — Rombach et al. (2022). Moved diffusion to a compressed latent space rather than pixel space, making generation fast and cheap enough for consumer use. This is Stable Diffusion.

Reasoning & Alignment

  1. Chain-of-Thought Prompting Elicits Reasoning in LLMs — Wei et al. (2022). Asking models to reason step-by-step dramatically improves performance on math and logic — but only above a certain scale. Foundation of all modern prompting methodology.
  2. Constitutional AI: Harmlessness from AI Feedback — Bai et al., Anthropic (2022). Replaced human labellers for harmlessness training with a model critiquing its own outputs against a written constitution. Made alignment training more scalable.
  3. Llama 2 Safety & RLHF Methodology — Meta (2023). Detailed, reproducible account of how RLHF and safety fine-tuning is applied at scale. One of the most transparent alignment papers from a major lab.
  4. Direct Preference Optimization (DPO) — Rafailov et al. (2023). Reformulated RLHF as a supervised learning problem, removing the need for a separate reward model and RL loop. Simpler, more stable, widely adopted.
  5. DeepSeek-R1: Incentivising Reasoning Capability in LLMs via RL — DeepSeek (2025). Showed that large-scale RL without supervised warm-up can produce strong reasoning in LLMs. Sparked significant debate about training efficiency and the role of SFT.

Efficiency & Fine-Tuning

  1. LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. (2021). Approximate weight updates as products of two small matrices, injected alongside frozen weights. Reduced trainable parameters by 10,000x. The default fine-tuning technique.
  2. QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al. (2023). Combined 4-bit quantisation with LoRA adapters, making it possible to fine-tune 65B parameter models on a single consumer GPU. Changed what was possible outside big labs.
  3. Scaling Laws for Neural Language Models — Kaplan et al. (2020). Showed that model performance follows predictable power laws as a function of compute, parameters, and data. Made training runs plannable and gave labs a roadmap for scaling.
  4. Training Compute-Optimal Large Language Models (Chinchilla) — Hoffmann et al. (2022). Revised Kaplan's scaling laws: most large models were undertrained relative to model size. Optimal training requires roughly 20 tokens per parameter. Rewrote how labs allocate training compute.

Retrieval, Agents & Systems

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al. (2020). Grounded language model generation in retrieved documents, dramatically reducing hallucination on knowledge-intensive tasks. The foundational RAG paper.
  2. ReAct: Synergising Reasoning and Acting in Language Models — Yao et al. (2022). Interleaved reasoning traces and actions in LLM outputs, enabling agents that plan and use tools iteratively. The conceptual foundation for most LLM agent frameworks.
  3. Reflexion: Language Agents with Verbal Reinforcement Learning — Shinn et al. (2023). Agents that reflect on past failures and refine their approach without gradient updates. Showed verbal self-correction improves task completion significantly.
  4. An Introduction to Vision-Language Models — Bordes et al. (2024). Comprehensive overview of how vision and language are jointly trained. Essential background for understanding multimodal models.

Multimodal & Vision

  1. An Image is Worth 16x16 Words (Vision Transformer / ViT) — Dosovitskiy et al. (2020). Applied the Transformer architecture to image patches. At scale, matched and then exceeded the best convolutional models, unifying vision and language architectures.
  2. Learning Transferable Visual Models from Natural Language Supervision (CLIP) — Radford et al. (2021). Trained on 400M image-text pairs from the internet using a contrastive objective. Zero-shot image classification, open-vocabulary retrieval, and the backbone of most multimodal systems.

Last updated: May 2026. Suggestions welcome — [email protected]