1) What the task is.
2) What solve % the top model gets right now.
3) What solve % a human would get.

Today, especially in many coding-related tasks, the era of the human baseline is over. We’re not going to be asking each other what a human gets on new benchmarks in these categories because it’ll be clear to everyone that a human, using a reasonable amount of time, would get 0% those coding benchmarks.

I think that ProgramBench was the first attempt at this new paradigm. It tasks agents with recreating an entire terminal program based just on its docs and interactions with the program itself, but with no ability to read or decompile any of its code. ProgramBench tasks include legendary programs like SQLite, FFmpeg, and the PHP interpreter.
These programs have been developed by large teams of engineers over decades. This benchmark isn’t just challenging, it’s clearly targeting a super-human ability. People initially thought it was too challenging (we got the same feedback for SWE-bench); but the rate of progress seems to imply that ProgramBench might’ve been not challenging enough.

One of the most important properties of a benchmark is how challenging it is. Benchmarks are used to direct model development: model builders iterate on training data, model architectures and optimization algorithms while checking the effects of changing those on benchmark results. Benchmarks set the north star for AI development. And, at least in coding, in many subdomains that north star should now be much beyond the abilities of any human.

SWE-bench was about solving bugs that would take a human engineer a few hours up to a few days to solve. SWE-bench has gotten pretty saturated at the frontier. We now have slightly tougher SWE-bench-like benchmarks, including DeepSWE and the TerminalBench 4 that have tasks that might take a human a week or two to solve. But those will also be saturated in the next few months. When a paradigm (SWE-bench in this case) gets saturated, that entire class of benchmarks gets saturated. It’s not enough to make a benchmark that is within the same paradigm and is 2x or 3x harder, that’ll just buy you a few more months. When a paradigm gets saturated, it’s time to move on to the next one.

And so if SWE-bench represents bugs that take a single human a few hours to days to solve, the next paradigm isn’t around tasks that would take an engineer a few weeks to solve; the next paradigm is around tasks that would take a team of engineers years or decades to solve.

Do I think that it’s a mistake to make a benchmark that represents tasks that humans can complete? No, absolutely not. Even at the frontier, even if we limit ourselves just to coding, models still lack human-level capabilities in a lot of subdomains. The vision capabilities in frontier multimodal models sometimes can’t count objects correctly and in general seem very weak (the newly relaunched SWE-bench Multimodal targets some of these capabilities; some top models solve just 40-50% of it right now).
Programming in less popular domains or using niche libraries can still be improved, and hallucination and context management are still an issue sometimes. AI seems to be automating many aspects of coding but every software engineer I know is working harder than ever. There’s still a lot of human-level stuff, in the SWE domain, that AI just can’t do. So there will be many more human-level benchmarks in the future. But I don’t think that many of the north stars for the AI community in the future will be human-level benchmarks.

Will future benchmarks be totally non-human, i.e. fully synthetic? No. SWE-bench was so obviously a good benchmark because it tasks agents with solving real bugs, from real, popular Python repositories on GitHub, that real humans solved at some point. Realness is what makes benchmarks challenging, it’s what makes them understandable both to model developers and model consumers, and it’s what makes models trained on these benchmarks useful in the real world. Being grounded in the real-world will continue to be super important in benchmarks. As benchmark developers, one of our most important tasks is to look at the real world and understand how we could arrange it in a way that would make for a good benchmark. This will continue! The best benchmarks will continue to be grounded, and will be built on top of real human data or environments or requirements.

Is it possible to build super-human benchmarks on top of real-world (human) data? Yes. The challenge is figuring out how to do it. ProgramBench takes real programs, some of which large teams of humans have built over decades, and asks agents to rebuild them in a single session. SWE-sweep takes a repo with real bugs, that were eventually all solved by many different contributors, and asks an agent to solve them all in one go. We’ll keep on finding these types of creative ways to take a human created artifact and turn it into a super-human testing/training ground.

And I think it will be super important to continue sourcing benchmarks from nature. As in, real stuff that real people have had to do to solve a real problem. If you ask a person to write a benchmark task for you, it’s probably not going to be very realistic, it’s probably not going to be very challenging, and it’s definitely not going to be at the super-human complexity level that the future paradigm requires. It’s because you get that person’s attention for a week or two of work; even if you have unlimited money you literally don’t have enough time to get a team of humans to work for a few decades to create something as complex as FFmpeg or SQLite.
Yes, there will still be a bunch of these purpose-built benchmarks as we continue to sweep away all the blemishes in sub-domains where frontier models are worse than humans; but in coding, I don’t think that these purpose-built benchmarks will not be the major north stars that direct the community.

Realness leads to scalability! A scalable benchmark is one like SWE-bench, ProgramBench or SWE-sweep, where new tasks can be automatically crawled from the internet with little to no human supervision, annotation or filtering. As mentioned previously, benchmarks set a north star for model developers to work towards; once a new north star is found we start working towards improving capabilities in that direction. And the best way to improve capabilities is to find training environments that are similar to the ones in the benchmark. If your benchmark was scalably sourced by crawling the web, your training data can be scalably sourced by crawling the web! And that makes improving your model in the direction of that benchmark much easier.

Imagine two competing model developers, one trying to hillclimb towards SWE-bench and one trying to hill climb towards TerminalBench. SWE-bench was sourced by crawling the web, so the first developer can, in a few days, crawl millions of SWE-bench-like issues from GitHub and with little manual labour have a substantial amount of working SWE-bench-like environments for use in training. TerminalBench tasks are ideated, constructed, audited by large teams of people over many weeks or months. So the second developer, that hillclimbs on TerminalBench, can’t just crawl training instances from the web. There’s not really any instances to be found on the web. This developer has to source, hire, train and supervise many teams of people to construct training instances, and not only is the cost high but the amount of time it takes is enormous and the potential for low-quality or bad instances is substantial.

In summary, scalable benchmarks come from scalable data sources which can be used for scalable training data generation which leads to improved model capabilities in a short amount of time, using little to no manual labor. This is true for human-level and superhuman-level tasks.

To conclude: in many subdomains of programming, a lot of our conversations going forward are going to shift from the past discourse of replicating human abilities, to now trying to build the future by training agents to do things humans will never be able to do. The jagged capability frontier makes this much harder for all of us to grasp. We’re going to have amazing capabilities in some narrow subdomains along with really bad performance in other subdomains, just within software engineering. Some of us will work on pushing performance on certain tasks from superhuman to supersuperhuman, while others will work on pushing performance on other tasks just to human-level. I wanted to write this blogpost to help me and others process this new beginning, this science-fiction role of us having to push forward a machine which can do some things much better than any of us can.

To push the AI frontier and make AI more useful for humans, we need to understand and then formalize the gap between ‘what agents can do’ and ‘what humans need’ via benchmark building. This is becoming harder than ever, but there’s definitely still a gap there so there’s work for us to do.

This is an amazing, stressful, confusing and inspiring time but not an unprecedented time. Horses can carry more weight than you or I, for longer distances, electric grids and radio networks transmit energy or information faster than any human could, and vaccines protect us in a way no human ever could. We’ll figure this out too.

My new coding benchmark checklist:

1. Is the task easy to explain? It should be explainable to a highschooler who has completed 1 programming class, in a short simple sentence. One of the goals of a benchmark is to explain to the wider community where models are in terms of capabilities; if your benchmark itself is too hard to explain it’ll be hard to use it to explain models. Beyond just the task, everything in the benchmark should be straightforward to explain, almost just obvious. Including the scoring/grading/metric, what the starting environment for each task is and what the directive given to the model is. The more complicated your benchmark is the harder it is to get it to go viral. Of course, the challenge here is to build a benchmark that is easy to explain to humans but hard for models to solve.

2. Minimize the benchmark-reality gap. How far away are the tasks in your benchmark from tasks that humans do in their day-to-day life or job? Would an agent that could do this task autonomously or semi-autonomously be useful to a large group of people? If you’re too niche that could lead to a harder time convincing people that your benchmark is useful. But if you’re too general the tasks will probably end up being too easy. Finding that balance is hard.

3. Is your benchmark currently hard for frontier models? Top accuracy should be <5%. There’s also a danger of making your benchmark too hard, but 99% of the time the mistake that people make is that they make their benchmark too easy.

4. Are the tasks robustly verifiable by a computer program? Don’t use LM-as-a-judge, human raters, or any type of subjective scoring. Benchmarks that can be run quickly and without human intervention enable model builders to iterate quickly.

5. Do you have enough tasks? This is a tricky one to define, but you should aim for at least 100 samples. This is extremely case-dependent but you should also make sure not to have too many benchmark samples, because then the benchmark will be too slow to run. As our benchmarks become more complex, the number of samples in them goes down. MMLU had more than 10k multiple choice questions, SWE-bench Verified had 500 programming tasks, ProgramBench had 200 tasks, SWE-sweep has 100 repos (which have 4k bugs within them), and the actual toughest current benchmark, the Millennium Prize Problems, only has 5 open tasks :)

6. Is test/train data creation for your benchmark scalable? Good benchmarks make model developers want to iterate on those abilities, by building pre-training and post-training environments and data. If your benchmark data creation pipeline is manual, that will make hill climbing your benchmark harder. One of the benefits of SWE-bench, ProgramBench and SWE-sweep is that the data pipelines for creating benchmark tasks are fully or almost-fully autonomous, meaning that creating training data for these benchmarks is also fully or almost-fully autonomous, enabling model builders to quickly hill climb these abilities.

7. The environment for running the benchmark is cheap and widely reproducible. If your benchmark requires 9 hours of CPU time that’s great. If it requires 1000 hours of raw hardware access to an AMD EPYC 9005 CPU, that makes reproducibility much harder. Of course if you limit yourself to just building benchmarks that can be ran for 50 cents per run, you’re not gonna get very far, but make sure your benchmarking environment is reproducible.

I’ve spent a lot of my career both on building benchmarks and on building systems that push forward the state-of-the-art on a given benchmark, and I believe that building good benchmarks is just as important as building new systems.

Designing a good benchmark is challenging and I’ve spent a lot of time recently thinking about what makes for a good benchmark. I’ve distilled it down to three main properties:

1. Natural:

Try to build a benchmark that has natural questions that some category of humans ask on a frequent basis. For example, the questions in our SWE-bench are made up of real bugs that users reported in popular GitHub repos. The task is to take the reported bug and the repo (as it was at the time the bug was reported) and try to fix the bug. That’s a very natural task that many people do on a daily basis (and even get paid for). Other natural tasks that we’ve recently turned into benchmarks include answering questions such as “What yoga studio near me has vinyasa classes before 8 AM on weekdays?” (see AssistantBench) and “Which paper first showed that transformer language models can’t extrapolate to long sequences?” (see CiteME).

Sometimes I see new benchmarks come out that don’t fulfill the naturalness criteria, and they often have a hard time getting excitement from the community. I find benchmarks that contain IQ test-like questions, where you have to identify patterns in diagrams to not be very exciting. Or any ‘common sense’-like benchmarks that have questions like ‘Bob threw an egg at Alice’s face. Is Alice happy, sad, or ambivalent?’ These types of benchmarks might have been interesting in the past, when our LMs were still struggling with basic tasks, but now that our language models are becoming more capable, we need to challenge them with tougher and more realistic tasks.

Another way to think about whether a benchmark fulfills the naturalness criteria is to evaluate whether it fulfills what I term the usefulness criteria: would a system that got better-than-baseline accuracy on this benchmark be useful to humans? Would it make anyone more productive? A system that can autonomously fix bugs would save lots of time for developers, even if it only managed to fix the easiest ten percent of bugs. A system that can quickly find me a yoga class that meets my needs would save me time.

I’ve also noticed that there are two simple indicators for a benchmark being unnatural, and so I try to avoid building benchmarks that have these properties:

A. The question set-up is unrealistic: For example, if a benchmark contains multiple choice questions, I believe it is unnatural. When I go to the doctor, I never say “Doctor doctor, my elbow hurts, and it is definitely happening because of one of these four options…”. Always think of your question set-up and if it seems unrealistic, try to modify it.

B. The questions are made up and not taken from actual questions asked by actual humans: If you work for Google and you’re tasked with building a challenging question-answering benchmark, a really nonoptimal thing to do would be to sit around by yourself in a room and just try to think of questions. You’d probably come up with weird questions that no real user would ever ask. A really smart thing to do would be to look at the Google Search logs and try to filter it to find questions that users entered and did not find a good answer to (for example, this might be indicated by the user going to the second result page or because the user spent more than five minutes on the initial results page).

SWE-bench contains real bug reports filed by real users on real GitHub repos. I think this makes the benchmark much more exciting to the community. Using questions that actual users asked implies that by building systems that get higher scores on the benchmark, we would be fulfilling a real-world need.

2. Automatically Evaluateable:

In a benchmark, given a model-generated answer to a question, we need to determine if the model was right or wrong. Sometimes this is easy, but depending on the question type, this could be hard or impossible. Validating the correctness of code could be a challenging task, since there are many different ways to program a given function, and that’s why benchmarks such as HumanEval and SWE-bench use unit tests to automatically validate code.

Summarization is a task that I think could be super useful for humans (“Write a 500 word summary of this patient’s medical file”) but we’ve seen very little development of new benchmarks in this space because evaluation is just so hard. There are many different ways to correctly summarize a given text, but evaluating these summaries is hard. Some have proposed using an LM to evaluate LM outputs but I don’t think that that’s the right way to go. We should either use an LM to solve a task, or use it to judge outputs, but if we use it as both the solver and the evaluator that leads to problems.

3. Challenging

If you launch an automatically evaluatable and natural benchmark, but the accuracy of the best LM at launch is 80%, people will see your benchmark as already being solved and won’t want to try and build models to improve performance on it. I think making your benchmarking challenging is critical. I think that at launch, a good benchmark should have the top LMs achieving between 1% to 35% accuracy on it.

Edit, January 2025: Due to the extremely fast development of LMs these days, I currently recommend that benchmark builders launch their benchmarks with the top accuracy being between 0.1% to 9%. Anything higher probably means that the benchmark is too easy.

Edit, May 2025: I had to make another edit. Due to the speed of development of AI I’m now asking my collaborators, not to think of benchmarks that would have AI systems achieving 0% at launch, but to think of benchmarks that would have systems achieving “-200%” at launch. Find questions that are so hard that even if the models improve 3x they’ll still get zero. Just building a benchmark where models get 0% today might not be enough anymore. You have to look at how the models have been improving over the past 3-6 months, try to predict where they’ll be in 6-12 months and build benchmarks that would not only make current models fail, but benchmarks that would make the models of next year fail as well. Anything easier than that might get saturated much more quickly than you expect.

If you find a benchmark idea and build it out and it’s natural and automatically evaluateable but you build a baseline and it gets 70% right, one thing you might want to consider doing is to use that baseline to filter-out the easier instances in the benchmark. For example, our Bamboogle benchmark had tough-to-answer 2-hop questions, and we built the dataset by filtering out all questions that Google Search answered correctly. For CiteME, we filtered out all questions that GPT-4o managed to answer correctly in a prompting-only setting (i.e. non-agentic). I think that building benchmarks by finding tasks that a strong existing approach can’t solve is a great way to go.

Edit, June 2026: If you are collecting benchmark tasks from a specific domain and find yourself needing to filter out some instances in order to make the benchmark harder, you are working with a task distribution that is too easy across the board. Sure, you can filter out some easier instances now, but that will only be a temporary measure. Your overall task distribution is too easy and you should probably move towards collecting tasks from a totally different and much harder domain.

Beware- researchers are humans and humans have emotions. If at launch, the top model’s accuracy is less than 10%, that might seem very intimidating for most researchers, and they might not want to work on your benchmark at all. Try to plan for that. For example, when we launched SWE-bench the top model’s accuracy was 1.96%. Almost everyone I talked to at the time was intimidated by this and didn’t want to approach it. I wasn’t worried, because we immediately started working on SWE-agent after releasing SWE-bench. I remember telling the team that if we got anywhere near 10% accuracy, the community would see that SWE-bench isn’t as impossible as it seemed, and that would get the ball rolling. Eventually we launched SWE-agent at around 13% accuracy and soon afterwards a barrage of other models appeared, each getting better accuracy than the previous one.

Bonus Property:

Building a benchmark that would be hard to leak into the training data is something that I think about all the time. Could we build a benchmark such that even if the benchmark itself leaks into an LM’s training data, it won’t really help that LM in getting a good score on the benchmark? In SciCode, we had PhDs write very tough programming challenges related to their field of study. Each instance in the dataset is a description of a function and the unit tests to validate whether the model programmed it correctly or not. We intentionally do not release any of the answers to these programming challenges, to make sure these answers are never inserted into any LMs training data. This way, even if our benchmark fully leaks into an LM’s training set, it still won’t be able to produce the right answers to the questions. Achieving this property is extremely difficult, and so it’s not something I try to do with every single benchmark I build.

Other guidelines:

1. Have one number for your benchmark. One metric that people go for. “We get 87% on HumanEval” is the vibe you are going for. Don’t have three metrics, like accuracy, precision, and recall, have just one. Don’t divide accuracy by category, have just an overall accuracy. This is really important. You want to make use of your benchmark as easy as possible. You want people to get it right away. If you start having seventeen metrics and nineteen categories it’s going to be complicated for people to understand what you’re trying to do, and that will lower the chances of your benchmark catching on.

   When you write the analysis section in your paper about your benchmark, it’s totally fine to present other metrics for each model, or to break down performance by category, but you should only do that there, and not have the categories or other metrics when you generally talk about the benchmark.

2. When you write a paper, always include very strong baselines, both based on strong proprietary models and on leading open source models. You should never try to make your benchmark look more impressive than it actually is by including only weak baselines, or baseline systems that use old or outdated models like GPT 3.5 or Llama 3 7B.

3. How many tasks should your benchmark have? I think a good minimum is to have at least 150 tasks in a benchmark. But defining a ‘task’ is hard, a benchmark that has 50 ‘tasks’ which each have 10 subtasks, could also work. Aim for 300-500 tasks so that your benchmark has more statistical resolution. In my experience, having more tasks than that doesn’t really help. I would also use the 500 task point as an upper limit, because having more tasks means it could take a very long time to run your benchmark, which could be a big downside for adoption.

4. Benchmarks typically get saturated within a year. And so when you build a benchmark, I don’t think it’s important to worry about questions like “Will this still be an interesting question in five years?”. Deep learning moves incredibly quickly, and there’s no way to predict where we’ll be in more than a year. For this reason, it’s also OK to have benchmarks that ask questions that will probably have totally different answers in a year or two, such as “Which yoga studios near Bushwick in NYC have a Vinyasa class before 7AM?”.

Benchmarks are great because they provide a lot of room for creativity, and they can be super impactful in guiding the community towards the future. I hope this post helps you in building the next big LM benchmark. And always remember- rules are meant to be broken. I do not think that a benchmark that does not follow all of these rules is bad. I just think that these guidelines are a good indicator for whether you’re on the right path or not.

Concluding Thoughts

Here are some questions to think about while you’re designing a new benchmark:

1. A benchmark is a collection of tasks, where each task is made up of <request, environment, stopping criteria, scorer> 4-tuples.
        A. The request is what you want the model to actually do, i.e. in SWE-bench it would be “Fix this issue “ + issue_text.
        B. The environment is a total description of the environment that the agent will act in while solving your request. Is internet access allowed? What dependencies are installed and which ones are not? Are there any special tools you will be providing the agent with?
        C. The stopping criteria is how you decide when to end an agent’s run. For some tasks the agent will probably issue a ‘submit’ command and exit but you need to decide how to act when that never happens. Are you going to have a turn limit per task? A cost limit? A walltime limit? A combination of these? All answers are viable, you just need to decide.
        D. The scorer takes the environment as it was when the agent exited and scores it. Will you build a binary pass/fail benchmark, like we did in SWE-bench with the fail2pass and pass2pass tests? Or will you build a benchmark with a continuous score, like we did in AlgoTune, where we ask agents to speed up computer programs, and the score per task is the agent’s code total runtime divided by our baseline’s total runtime. Or will you use ELO like we did in CodeClash? There are many possiblities here.

2. What is the baseline scaffolding that you will use and how similar is it to the best scaffolding in common use right now? For example, if you’re asking coding questions, and your scaffolding doesn’t allow for code execution, that’s not a very good representation of reality. If you’re asking knowledge questions and don’t allow access to the internet, that’s not realistic. Try to make your scaffolding as close as good as you can. This frequently doesn’t take much effort as people think. mini-SWE-agent is able to get very competitive scores (and sometimes even surpass) Claude Code these days, even though it is orders of magnitude simpler. I talk a lot about how much easier it is to sell a benchmark that is realistic, and part of that is making the tasks realistic, but you should also make your baseline scaffolding realistic, otherwise people will mistrust your results.

Benchmarks are what moves the frontier of AI forward, there isn’t anything more important than building good new benchmarks. Good luck!

This post contains five patterns that I’ve noticed that both my peers and I use to think of new ideas for research projects. It’s important to note that this advice might only apply to the research sub-field that I’m in and might only apply to people who want to perform research in a manner that is similar to the one I use. There are many ways to succeed, some of which are orthogonal to the guidelines that I use.

1. Focus first on finding a problem, not a solution.

A common mistake I see people do is start a project by thinking about a model they want to build, without even thinking about whether it’s even necessary. They’ll get really excited about a certain method and try to just build something based on that. “Let’s build a retrieval-augmented language model!”. “I want to build an LM agent that uses reinforcement learning!”. But I don’t really think that research can be done with such a vague goal. In order to ground your research, I believe that it’s incredibly useful to find a specific issue that the community would care about, and then work from there. So for example “I’ve noticed that GPT has a really hard time answering questions that have a spatial component” or “I’ve noticed that GPT has a really hard time programming solutions to programming puzzles about graphs” are good starting points. First define an issue, then try to figure out if it would interest the community, then build a dataset of example problems and then try to build a model that solves them. While building a solution, if you figure out that you do need RL or retrieval, then integrate that into your solution. But you’ll always be grounded in the results you’ll get from running your various baselines and new models on the dataset that you built. That’ll tell you whether those components are actually improving your model or not. I used to think that the majority of the work for researchers in deep learning was in building solutions, but I’m now pretty convinced that the majority of the work is in finding good problems!

2. Once you find a good problem to work on, it’s better to iterate and experiment through many potential solutions that are maybe good rather than working through one or two solutions that are “definitely” correct.

The king of deep learning is empirical results. Once you’ve decided which problem to focus on, the only thing that you can do if you want to test a certain solution is to run it and see what happens. Therefore, for me it has been very beneficial to iterate through the idea-brainstorming-to implementation-to-results cycle many times per project. I sometimes see people get stuck in endless pontification and over-analysis of ideas without ever opening an IDE. They’ll think of an idea and then analyze it on a whiteboard for weeks before they even consider sitting down to program it. In my view, that hypothetical analysis is not very useful. Since there’s not really a theory of deep learning yet, the utility of analyzing potential solutions on a whiteboard for more than a few hours is not much. As ML practitioners, our source of truth is the results we get after running an experiment. Yes, you shouldn’t just program every single idea that comes into your head, but also, once you think of something and spend an hour or two thinking it through, just implement it and see how it does.

Only many empirical experiments will build an intuition for what works and what doesn’t and will help you define the path forward. Quick iteration through a trajectory of research ideas leads to progress in deep learning. I’ve also observed that one of the most important factors is the number of iterations on research ideas and not the magnitude of each idea. So I generally recommend thinking of and working on solutions that are as simple as possible, so that you can go through the idea-brainstorming-to implementation-to-results cycle as many times as possible per project. Understanding what ideas to implement and which ones not to, how to prioritize different possible directions, and how to reject ideas before even implementing them are skills that you will learn as you spend more time doing research.

In the second year of my PhD I had an idea for a new type of retrieval-augmented LM. This model had two components, a retrieval and an LM. The LM was pretty much off-the-shelf, and the retriever was the component that I wanted to innovate on. The full model was quite complex and so the first prototype took me about three months to build. I then ran it and it didn’t improve performance. So I then thought of a second version, which was also quite complex and ended up taking me two months to program. That model didn’t work either.

I then realized that I should first verify if this idea could even work by using an oracle retriever: basically cheating at the retrieval stage to make your retriever as good as it could be. This oracle-based model took a few days to program. When I ran it, it also didn’t work, thereby making my confidence in the overall idea tank. If the model couldn’t work with an impossibly-powerful (enabled by cheating) retriever, I didn’t think it could work with a less powerful, but possible, retriever. In hindsight, when I had the original idea, I wish I would’ve been able to notice that the initial prototype would’ve taken three months to program, and instead of starting from it, it would’ve been much smarter to start from the much smaller oracle prototype, and only successively build my way up to the three-month prototype, only if the earlier prototypes showed promise. That would’ve saved me a lot of time.

3. Write a paper that many people would want to read:

Your paper should either teach us something new about an existing system, method or benchmark, or achieve better performance on an existing benchmark, or present a new benchmark. Good papers sometimes do two of those things.

What does it mean to achieve better performance?

If I have a model that can get 70% accuracy on SWE-bench, and you develop a new one that achieves 72% accuracy but is three times slower then that’s probably not an interesting new system. Improving performance doesn’t just mean improving accuracy and ignoring all other factors. When we look at a system, we have to observe both accuracy but also training time, inference time, memory usage, latency and disk space usage. For a new system to be better than an old one, it should improve on one of these metrics while keeping the other ones at the same level or better. If the improvement is substantial, it’s fine to disregard this rule, and for example, present a system that achieves 99% accuracy on SWE-bench while being 30% slower than the current state-of-the-art. But in that case, make sure to compare your method to the baseline, when the baseline is given 30% more time. You should always be comparing your methods to the strongest baselines you can find, and you should strive to make that comparison as fair as possible.

Why is community excitement important?

You should definitely work on research that excites you, but you should also try to find a topic that would also excite a large amount of the other researchers in the field. Papers only reach a small percentage of their target audience. If you’re writing a paper about a niche topic that only has 50 researchers working on it, maybe 15 of them would actually hear about your work, 5 of them would read it and likely none of them would perform follow-up work. I therefore find it incredibly useful and rewarding to work on topics that have a wide interest in the community.

4. Write a paper that would be an interesting part of the research discourse that will be happening in nine months.

In my experience, for my style of work, research projects take nine months on average. That means that when you start thinking of ideas you shouldn’t think of ideas that would be interesting if they were published today or tomorrow, because it’s going to take you much longer than that to write a paper. You should definitely make sure that you’re not writing a paper that would have been relevant one or two years ago. One simple rule of thumb is to make sure that at least some of the related works that you’re citing are from the past year or two. If your latest citation is from five years ago, there’s a big chance that you may be writing a paper that would be irrelevant to the current research discourse.

The most important skill to learn here is observing the trajectories that occur within your research sub-community and being able to predict where they will go. As you do research for more time you’ll get better at predicting where your research field is heading and how to do work that will fit into that puzzle as well as possible. It sounds impossible at first but I believe that, at least in deep learning, it’s possible to predict with high accuracy where the field will be in 9 months. It’s totally impossible to predict where the field will be in 18 months or longer, and that’s why I recommend not working on projects that would take that long.

5. Keep it simple.

And what do I mean by ‘it’? Everything. Try to work on problems that are easy to describe. Try to find simple solutions for those problems. Try to describe your solution in your paper in as simple of a way as possible. Try to write the code for your method in a simple way, and make it easy for others to run and extend your code.

Think of the counterexample here. If you worked on a problem that took two pages to describe in a paper, would any reader stick around for that whole description, let alone would anyone stick around to read about your solution? I also feel like most of the most important problems in our field can be stated in a sentence, so if it takes you six paragraphs to describe the problem you’re working on, that might be a hint that you’re working on a problem that is too niche or contrived.

It makes me happy when lots of people get excited by the work I release. Simplicity is one of the main driving factors in finding ideas that lots of people might get excited by. If you work on a super complex topic, there’s a high chance that very few people would even understand the problem you’re trying to solve, let alone your solution.

As for keeping your solution simple- the ML community has proven time and time again that the best methods that have the most long-lasting impact are always the simplest ones.

As with all rules, in some cases it does make sense to violate this rule. Tim Dettmers’ bitsandbytes is a super popular efficiency library. It’s made up of a lot of very non-simple CUDA code. But everything else about this library is extremely simple: it’s really easy to use, and the motivation behind it is very clear (“bitsandbytes helps you run big models on small GPUs”).

As a researcher that has to publish, it may be tempting to find complex solutions to complex problems, since reviewers are frequently impressed when they see many equations and proofs in a paper. But in my experience, while those types of papers may initially get accepted, over time, complex solutions do not have much lasting impact. Complex papers are harder to read, and their code is usually harder to extend; these properties substantially harm impact.

Closing note:

The strength of the research community is in its large size and diversity. There are many ways to do good research, some of which align with the tips above and some of which don’t. I hope that by sharing these lessons that I’ve learned over the years I’ve helped you improve your ability to do the best research you can.

If you enjoyed this post, you might also enjoy my post on tips for junior researchers, which focuses on how to do research and how to work with a mentor.

Thank you to Nelson Liu, Will Merrill, Samuel Ainsworth, Shunyu Yao and Naomi Saphra for comments on previous drafts of this post.

Acknowledgments

When Noah A. Smith accepted me into the Ph.D. program, he invited me into an environment that eventually produced a new and improved me: a version of me that is more open to new ideas, better at executing, smarter, and more patient. I will be forever grateful to Noah for believing in me and fully trusting me from day one and for always treating me with respect, patience, and love. Noah not only taught me how to do science, he also significantly improved my storytelling abilities, my ability to hold onto a reader’s attention, and my ability to frame my stories in an appealing way.

Although formally Mike Lewis wasn’t on my committee, he was on all of the papers in my thesis, and I consider him to have been my de facto co-advisor. Our ability to bounce ideas off of each other while improving them at each iteration is incredible.
Mike’s advising complemented my thinking style in a way that immensely improved the level of the work I did during the Ph.D. Working with Mike has shown me how much easier it is to tackle complex problems when you approach them with the right partner.

The end of my first academic year was rough for me, both academically and personally. It was then that Omer Levy invited me to be his intern at Facebook AI Research (FAIR). This totally changed the course of my Ph.D., and I am very grateful to Omer for that opportunity. Omer taught me many important skills for empirical machine learning research that I still carry to this day.

After spending half a year at FAIR as Omer’s intern, Luke Zettlemoyer invited me to stay at FAIR, as a visiting researcher, for two additional years. That allowed me to work with Mike and do resource-intensive research that would not have been possible without an industry affiliation. Throughout this time, Luke gave me total freedom to explore and do research on whatever I wanted to and always provided support. Being at FAIR for two and a half years made my research much stronger than it would have been if I hadn’t been there.

I’m grateful to Jonathan Frankle for inviting me to join MosaicML as a visiting researcher for six months and for always supporting my research.

I’m grateful to Kyunghyun Cho for hosting me at the wonderful Center for Data Science at NYU where I spent the last academic year of my PhD.

I’m grateful for my first mentee, Muru Zhang, for showing me how rewarding it can be to advise.

I’m grateful for the help of Elise deGoede Dorough and Sandy Kaplan from the Allen School at the University of Washington.

I’m grateful for all the other collaborators I’ve had during my Ph.D., who I learned so much from: Adi Haviv, Ori Ram, Peter Izsak, Sewon Min, Ludwig Schmidt, Will Merrill, Alisa Liu, and members of the BigScience project.

I’m grateful for my friend Samuel K. Ainsworth. I met Sam at visit days before starting graduate school where we bonded over our love for free food. Five years later we’re still eating free food together, hopefully for many more years.

I’m grateful for my friend Ivan Evtimov. We met during the beginning of the first year of school since Ivan would always show up to the board-game nights I organized, and we have been friends ever since.

I’m grateful for my friendship with Tim Dettmers and Gabriel Ilharco and for their smart comments and strong support for my work.

I’m grateful to my academic twin brother Jungo Kasai for five years of conversations, comments on papers, and reminders about administrative tasks that I had to complete.

I’m grateful to Ian Covert, Edward Misback, and Nathan Hatch for our countless days together playing tennis.

I’m grateful for my friends back home Ofek Doitch, Dean Stephansky, Andrey Shulika, Nimrod Fiat, Gili Yablonka, Nir Aviv, Tzvika Geft, Shai Kazaz, Gil Levi, Gregory Axler, and Lior Uzan.

I’m grateful to Ben K., Sammy, Danny, Kyra, Dasha, Ben A., Jamie, Paul, Jason, Lana, Carolyn, Shirley, Ivan, Pearly, Adi, Orville, Joe, Amitai, and everyone else who taught me how to stand on the shoulders of giants.

I’m grateful for James, Trixie, Andrea, Raven, Grini, Ian, Za, and everyone else who went to Brazil with me.

I’m grateful for my therapist who has taught me how to better understand myself and others.

I’m grateful for all the love and support I’ve received from Saba, Safta, Ima, Ori, Avia, Yossi, Liat, Idan, Ronnie, and Bara, without which I would not have been able to complete my Ph.D.

Dedication

Dedicated to my grandfather, Haim Yehiel (born in 1926 in Thessaloniki, Greece, died in 2022 in Ramat Hasharon, Israel), for inspiring me to be a nerd.

This is him at the Technion in Haifa in 1945 during an undergraduate class on welding.

Imagine you’re building the next version of a causal code prediction model like Codex. When we train an LM like this, due to GPU memory limitations, we must pick a finite sequence length, say 4,000 tokens, to train the model on. If at inference time, users only want to make predictions in code files shorter than 4,000 tokens, we’re good. But if a user wants to make a prediction for token 4,001, it would be impossible with absolute position embeddings. If you use learned position embeddings, feeding 4,001 tokens to your LM will simply throw a runtime exception (there is no 4,001 position representation). If you use sinusoidal position embeddings, the model will run given 4,001 inputs, but as we show in the ALiBi paper, it will produce really bad predictions (for any token beyond the first 4,000).

Relative positional methods like ALiBi solve this. The T5 bias is another good option, although personally I prefer ALiBi because our paper shows it obtains better results and also it’s faster and doesn’t require any trainable parameters.

The rotary method has shown some strong results when evaluating sequences that are shorter than or equal to train length, but in our paper we show that it is not able to extrapolate to longer sequences. In addition, it is slower than ALiBi and the absolute positional methods. Lastly, while some people consider it a relative position embedding method, in my opinion, that’s incorrect. Rotary simply element-wise multiplies position representations by the word representations (instead of adding position reps to word reps, as is done in the absolute methods). This means that Rotary still employs position embeddings, which in my view makes it an absolute position method, not a relative one. This thread has more details on why I believe absolute position methods are not the way to go.

Can’t I just use an absolute position method and a sliding window to extrapolate? Short answer: Depending on how you implement this, it either won’t work or will be very very inefficient.

Details: Absolute position embeddings are battle tested and so when engineers want to build LMs that can handle longer sequences, one of the first ideas they have is to use a sliding window with an absolute position embedding method. So if we go back to our Codex example from before, we would train the same 4,000 token LM, but at inference, we would limit the attention sublayer to only attend to the last 4,000 tokens. So when we input 4,001 tokens we would only attend to tokens 2 to 4,001 and when we input 4,002 tokens we would only attend to tokens 3 to 4,002 and so on.

There are two ways to do this:

1. The simpler approach is just to re-encode everything at every timestep. So in the first feedforward pass of the LM we encode the first 4,000 words, and then in the second feedword pass when we’re looking at words 2 to 4,0001, we discard everything from before and re-encode everything even though there’s an overlap of 3,999 of the words between the two runs. This will work, but is very inefficient. You have to re-encode everything during each forward pass beyond the first 4,000.
2. In the second approach, we don’t re-encode previously encoded tokens. This will just lead to really bad predictions (I’ve tested this). This is because the same token will be assigned to different positions during subsequent inference runs, which means that its cached representation is invalid. See visualization below.

In both of these approaches, when we’re not extrapolating (i.e. when we’re doing inference on tokens 1 to 4,000), we do normal LM inference. So for example, when token 600 comes in, we have already computed representations for tokens 1 to 599, so we attend to those representations and only have to construct new outputs for token 600. As mentioned above, if we don’t use a relative position method like ALiBi, continuing this inference beyond the first 4,000 tokens will either just produce really bad predictions or it could work but very slowly, if we re-encode the past tokens. Using ALiBi means we get to continue doing inference much beyond token 4,000 without needing to re-encode anything.

To visualize approach 2 from above, I have a toy input sentence here with a toy language model, whose context size is 4 tokens. We see two subsequent inference passes.

Let’s look at the token ‘jumped’ in these two subsequent forward passes with the sliding window + absolute embeddings approach. ‘Jumped’ was assigned position 4 in the initial forward pass, and then if we use this sliding window approach we would have to treat ‘jumped’ as position 3 in the next forward pass, even though we need to attend to the old cached representation in which it had position 4. This weirdness that the model definitely didn’t experience during training just leads it to produce very bad predictions. This approach does not work.

Edit: Shortly after I wrote this blog post I was made aware of this new paper which reveals new evidence showing the weakness of absolute position embeddings.

To learn more about ALiBi and relative position embeddings in general, watch my lecture here:

1. Research is hard and involves a tremendous amount of failure. It’s totally normal to feel like you didn’t progress in the past week, month, or year. When you read a paper, the finding is frequently presented as an obvious step that was the result of a seemingly short exploration, but when you actually do research you realize that good results never just fall into your hands. They’re the results of many months if not years of failed expeditions and many experiments that did not work. Failure seems extremely demoralizing at first but experienced researchers will attest that failure isn’t that bad- you learn a lot from failing (you learn what not to do, or what doesn’t work) and after enough failure, you achieve enough understanding to have a good enough idea for something that will work. Even as a senior Ph.D. candidate, I will still sometimes “fail” for months on end, running experiments and exploring ideas that lead to nothing interesting. It’s still tough for me to go through this and while I’m in that zone of trying things and constantly failing it feels depressing. But at some point things start working and it makes all that failure worthwhile.
2. Don’t ever lie, make up results or sweep negative findings under the rug. These things might help you in the short term but in the long term they never will, and good research is all about the long term. For example, if you run your new model with 4 different random seeds, and in one of those runs the improvement is 10%, and in the other three runs the improvements are -2%, 1% and -9%, that means that the 10% run was a fluke. If you wanted to, you could submit a paper where you don’t mention the other runs, but that would be deceitful. Sure, that paper might get accepted, but eventually someone will try your idea, and they’ll probably run it with a few different random seeds, and notice that your improvement is not statistically significant. Not only will they then not use your method, they’ll also be wary of your future papers. Junior researchers are anxious to get an initial publishable result and might put aside annoying things like statistical significance, but in the long run that will hurt them. There’s no rush- good research takes time, and it’s better to take one year to write a solid paper than it is to write four low-quality papers that each took three months to write.
3. Focusing is important, especially in the beginning. I’ve noticed that some junior researchers are afraid to pick one project and stick to it, instead preferring to simultaneously work multiple projects. Their explanation is that they’re not sure which of the projects will pan out, so they want to try multiple projects at once so they have a better chance of one of the projects making it big. In my experience this logic is flawed. Good research requires intense focus and if you work on three projects at once you’re not really going to focus on any of them and so the outcome will probably not be as good as it could’ve been. Instead, I recommend picking the one direction that you’re most excited about and focusing on that. You’ll see that as you keep working on it, it will evolve, pivot, and change many times into many different directions. You may even get exhausted at some point and want to take a small break to work on something else, before coming to back to the original direction. That’s ok. Just try to, at least for your first year in grad school, work on one direction at a time. Once you’ve submitted your first paper, you’ll be much wiser and have a much better understanding of the academic idea-to-submission cycle, and then if you feel towards your later years that you’d like to work on two directions at once, I believe that you’ll understand what that means and be able to handle that.

Working with a mentor:

1. It’s ok (and even recommended) to say “I don’t understand you”. When we start doing research we usually do it with an advisor who is much more senior than us, a professor if you’re in grad school or a PhD student if you’re an undergrad. Sometimes the advisor will say something super complicated that is totally incomprehensible to the junior person. When this happened to me as a junior researcher I sometimes was too afraid to say “I don’t understand”, since I thought that the senior person would think that I’m dumb. Now I know that we all think in different ways and something that’s obvious to us might be hard to explain to a different person. Good researchers understand this and are very open to explaining and re-explaining and re-re-explaining their thoughts. Good senior researchers also know that explaining new ideas to different people helps us to better frame our thoughts and understand how to write them in a paper.
2. It’s ok (and even recommended) to say “I don’t know”. We’re all doing research since we don’t know a lot of things and we’re trying to figure them out together. Saying “I don’t know” when you don’t know something doesn’t make you seem stupid – it makes you seem honest. If you just start pretending to know things you don’t and have answers for the topics you don’t have answers to, it’s not going to be very constructive.
3. It’s ok (and even recommended) to say “I don’t agree with you”. Progress in research is partially driven by disagreements. At any given moment there are many different paths being explored to solve each issue, and that’s how we progress towards the solution. If everyone worked on the same ideas that would be horrible. So disagreement (even within the same research group or mentor-mentee pair, or even with yourself, over time) is totally acceptable in the research world. Just be nice about it! Never say something like “that’s a dumb idea” or anything even close to that. But if you disagree with a certain direction or idea, find a respectful way to voice your concern. Doing this will let the other person try to convince you why they believe their idea is good, which is helpful for both you (now you understand what they want to do and why they believe in it) and them (you might have uncovered a potential weakness which they can now try to remedy).
4. Your advisor doesn’t have all the answers. Doing research is a multi-faceted endeavor, with many different questions to answer: What problems are currently relevant and exciting to the community? Which of these is the best fit for me? What is the high-level plan for solving this problem? How can I best execute that plan (implementing the model, figuring out what/where to find hardware, and so on…)? Once I’ve found a solution, what is the best framing for it? How can I best market my paper? Your advisor is going to help you with some of these questions, but might not be able to give you all the answers. Some of them will have to come from you- but one thing that might significantly help is finding another senior collaborator. If you look at papers coming out of the UW NLP (and many other ML/NLP/Vision) groups you’ll notice that a lot of them have both a professor co-author and a co-author who is a postdoc or research scientist (in addition to the main author, who is usually a PhD student). I’ve found this advising style to be incredibly useful: the professor provides high-level feedback on the story and framing, while the postdoc/research scientist can provide more low-level feedback about code and other implementation details.

Closing note

After reading an earlier draft of this post one of my friends said that I didn’t mention the most important part of succeeding in research- being lucky enough to find good mentors to work with. I’m incredibly fortunate to have been able to work with some of the nicest and smartest people in the world and unfortunately not everyone is this lucky. Some mentors won’t let you say “I don’t know”, won’t respect your opinions when brainstorming, won’t give you the freedom to work on things that interest and spark joy in you and will pressure you to meet made-up deadlines. I hope that anyone with a toxic mentor can read this post, realize that there are better alternatives, and try to seek one that would work better for them.

Thank you Noah A. Smith, Gregory Axler, Gabriel Ilharco, Mitchell Wortsman, Samuel Ainsworth, Ori Press, and Gabriel Stanovsky for comments on previous drafts of this post.

Bamboogle is a dataset that we constructed, made up only of questions that Google answers incorrectly. The leaderboard for it is here.

In our Compositionality Gap paper, we show that language models also struggle with these questions and that our self-ask prompting method substantially improves the ability of language models to answer these questions (better than Chain-of-Thought).

For more details, check out the video above.

Bamboogle was introduced in our Compositionality Gap paper which can be found here, and the dataset itself is here.

As language models grow in size they know more, but do they get better at reasoning? To test GPT-3, we generated lots of questions such as “What is the calling code of the birthplace of Adele?”. We show that as GPT-3 size grows, it does not improve its reasoning abilities on these types of questions.

Compositional Celebrities

To test the reasoning abilities of LMs we built Compositional Celebrities, a dataset of 8.6k questions in 17 different categories.

These questions all require first retreiving 2 facts and then conducting some basic reasoning about them. For example, to answer “What is the calling code of the birthplace of Adele?” a model must first know that Adele was born in the UK and then should figure out that it needs to return the calling code of the UK- +44.

The reasoning required to answer them is simple, and the basic facts are commonly appearing (since they are either related to celebrities or their birth country or year), but we believe that these 2-hop questions have never previously appeared in any text that could be in the training set of an LM.

The Compositionality Gap

We can check the accuracy on these compositional questions (blue) or the accuracy for each pair of sub-questions separately (i.e. “What is the birthplace of Adele?” & “What is the calling code of the U.K.?”).

The surprising result we uncovered is that the compositionality gap doesn’t narrow with scale!

The compositionality gap is the fraction of compositional questions that GPT-3 can’t answer even though it can separately answer the two sub-questions that make up the compositional question.

As GPT-3 gets larger it’s remembering more facts but it’s not able to compose ~40% of these fact pairs, at all model sizes between 1B to 175B parameters! Maybe scale can’t solve everything?

This surprising result also occurs in the InstructGPT-3 family of models! The compositionality gap stays around 40% no matter how much we increase model size.

In the table below, we zoom in on the results for the best GPT-3 model, davinci-002. Some compositional question categories are really easy for it, like Birthplace/Domain Name (80% acc), but some are super hard, like Birth Year/Lit. Nobel Winner (1% acc). We’re not quite sure why.

And finally, the figure below presents an interesting finding: when GPT-3 (davinci-002) is very confident about two facts, it will be able to answer the compositional 2-hop question about them with much higher probability!

Our paper is available here, and the Compositional Celebrities dataset is available on GitHub.

Self-ask is a new prompting method which improves the ability of language models to answer complex questions.

Normally a question answering prompt looks like this:

Question: Who lived longer, Muhammad Ali or Alan Turing?
Answer: Muhammad Ali 

Question: When was the founder of craigslist born?
Answer: December 6, 1952

Question: Who was the maternal grandfather of George Washington?
Answer: Joseph Ball 

Question: Are both the directors of Jaws and Casino Royale from the same country? 
Answer: No

In self-ask, we first have the model generate and then answer sub-questions about the main input question, before answering the input question. So our prompt would look like so:

Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali 

Question: When was the founder of craigslist born?
Are follow up questions needed here: Yes.
Follow up: Who was the founder of craigslist?
Intermediate answer: Craigslist was founded by Craig Newmark.
Follow up: When was Craig Newmark born?
Intermediate answer: Craig Newmark was born on December 6, 1952.
So the final answer is: December 6, 1952

Question: Who was the maternal grandfather of George Washington?
Are follow up questions needed here: Yes.
Follow up: Who was the mother of George Washington?
Intermediate answer: The mother of George Washington was Mary Ball Washington.
Follow up: Who was the father of Mary Ball Washington?
Intermediate answer: The father of Mary Ball Washington was Joseph Ball.
So the final answer is: Joseph Ball 

Question: Are both the directors of Jaws and Casino Royale from the same country? 
Are follow up questions needed here: Yes. 
Follow up: Who is the director of Jaws? 
Intermediate Answer: The director of Jaws is Steven Spielberg. 
Follow up: Where is Steven Spielberg from? 
Intermediate Answer: The United States. 
Follow up: Who is the director of Casino Royale? 
Intermediate Answer: The director of Casino Royale is Martin Campbell. 
Follow up: Where is Martin Campbell from? 
Intermediate Answer: New Zealand. 
So the final answer is: No

This leads the model to solve test-time questions by first answering subquestions about them, and this leads to an increase in performance.

The structure of our prompt allows us to easily parse-out these subquestions and have Google Search answer them instead of the LM. We show that this further improves performance. In our paper we call this system Self-ask + Search Engine. Note that Google Search does not have a publicly available API, and so we use SerpApi, which is a cloud service that provides an easy to use API to Google Search.

Watch our demo (embedded above) for a deeper overview of Self-ask and Self-ask + Google Search.

To learn more about the Compositional Celebrities dataset that we created to evaluate self-ask, and about the surprising compositionality gap that we discovered, go to this post.

To learn more about the Bamboogle dataset that we created, made up only of questions that Google can’t answer, go here.

Our paper is available here, and our code for Self-ask + Google Search is on GitHub.

The following video provies an in-depth overview of all of the topics listed above:

Is this interleaved pattern the best way to order these sublayers?

In this post, I’ll explain how we recently found a better way to order these sublayers. That ordering leads to superior performance on multiple language modeling benchmarks.

We started by generating random transformer models, varying the number of each type of sublayer, and their ordering, while keeping the overall model size (number of parameters) constant. Here are a few of these randomly generated models:

(Note that one self-attention sublayer has half the parameters of a feedforward sublayer, so you’ll notice that models that have more feedforward sublayers are shallower. )

We trained these models on the standard WikiText-103 word-level language modeling benchmark. While most of these randomly generated models performed worse than the interleaved model, about a third of these random models outperformed it (mostly by a small margin). Our analysis shows that models with more self-attention toward the bottom and more feedforward sublayers toward the top tend to perform better in general:

We also observed that models that had an equal number of self-attention and feedforward sublayers tended to perform better than models that had an unequal number of self-attention and feedforward sublayers. Based on this insight, we design a new family of transformer models that follow a distinct sublayer ordering pattern: sandwich transformers. Sandwich transformers are made up of n self-attention sublayers, followed by the regular interleaved transformer pattern, followed by n feedforward sublayers:

Our experiments demonstrate that a sandwich transformer outperforms the baseline interleaved transformer model. This result is made more interesting by the fact that our sandwich transformer is simply a reordering of the sublayers in the baseline model, and does not require more parameters, memory, or training time. On WikiText-103, sandwich transforms improve perplexity while also reducing the variance caused by selecting different random seeds:

Finally, we demonstrate that even though the sandwich transformer is motivated by random search experiments on WikiText-103, it can improve performance on additional domains and tasks. Sandwich transformers achieve state-of-the-art results on the enwik8 character-level language modeling dataset and on an additional word-level corpus. We conjecture that tuning transformer reorderings to specific tasks could yield even larger gains, and that further exploration of the ordering space may provide universally beneficial patterns.

Other conclusions and insights:

• The transformer layer is not the smallest indivisible unit. The self-attention or feedforward sublayers can each function independently.
• The transformer architecture is quite robust to sublayer order changes. A non-insignificant amount of the random orderings that we trained performed just as well (and sometimes better than) the baseline.
• The ‘extreme standwich’ ordering s¹⁶f¹⁶ (shown below) works almost as well as the baseline on WikiText-103.

• The optimal transformer ordering is not identical across different datasets. For example, the best sandwiching coefficient for WikiText-103 is 6, but the best coefficient for the Toronto Book Corpus language modeling dataset is 7. For character level language modeling the optimal sandwiching coefficients were also different.

The paper is available here. We also have a video presentation available here.