This repository contains a collection of papers and resources on Reasoning in Large Language Models.
For more details, please refer to Towards Reasoning in Large Language Models: A Survey
Feel free to let me know the missing papers (issue or pull request).
Contributor: Jie Huang @UIUC
Thank Kevin Chen-Chuan Chang @UIUC, Jason Wei @Google Brain, Denny Zhou @Google Brain for insightful discussions and suggestions.
Jie Huang, Kevin Chen-Chuan Chang
Emergent Abilities of Large Language Models 15 Jun 2022
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus
Language Model Cascades 21 Jul 2022
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, Charles Sutton
How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources 11 Dec 2022
Yao Fu, Hao Peng, Tushar Shot
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Huajun Chen
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, Kai-Wei Chang
A Survey for In-context Learning 31 Dec 2022
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, Zhifang Sui
Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, Erik Cambria
Nature Language Reasoning, A Survey 26 Mar 2023
Fei Yu, Hongbo Zhang, Benyou Wang
We mainly focus on techniques that are applicable to improving or eliciting "reasoning" in large language models like GPT-3 (175B)
Papers in this paradigm vary a lot and are usually based on small models trained on specific datasets. We list several papers here for reference (that is, the list is not complete). Please refer to our survey for some discussion.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, Richard Socher
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge 11 Jun 2020
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, Jonathan Berant
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, Augustus Odena
Soumya Sanyal, Harman Singh, Xiang Ren
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou
Boshi Wang, Xiang Deng, Huan Sun
Large Language Models are Zero-Shot Reasoners 24 May 2022
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa
Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models 16 Sep 2022
Ben Prystawski, Paul Thibodeau, Noah Goodman
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, Jason Wei
Wenhu Chen
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, Graham Neubig
PaL: Program-Aided Language Model 18 Nov 2022
Luyu Gao*, Aman Madaan*, Shuyan Zhou*, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, Graham Neubig
Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks 22 Nov 2022
Wenhu Chen, Xueguang Ma, Xinyi Wang, William W. Cohen
Hangfeng He, Hongming Zhang, Dan Roth
Training Verifiers to Solve Math Word Problems 27 Oct 2021
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, Weizhu Chen
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, Tushar Khot
Zhuosheng Zhang, Aston Zhang, Mu Li, Alex Smola
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, Hanie Sedghi
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, Jun Zhao
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, Ed Chi
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, Denny Zhou
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, Ashish Sabharwal
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis
Dheeru Dua, Shivanshu Gupta, Sameer Singh, Matt Gardner
Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning 31 Jan 2023
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, Yongbin Li
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents 18 Jan 2022
Wenlong Huang, Pieter Abbeel, Deepak Pathak, Igor Mordatch
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning 19 May 2022
Antonia Creswell, Murray Shanahan, Irina Higgins
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi
Faithful Reasoning Using Large Language Models 30 Aug 2022
Antonia Creswell, Murray Shanahan
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering 20 Sep 2022
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, Ashwin Kalyan
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, Xifeng Yan
Distilling Multi-Step Reasoning Capabilities of Large Language Models into Smaller Models via Semantic Decompositions 1 Dec 2022
Kumar Shridhar, Alessandro Stolfo, Mrinmaya Sachan
Teaching Small Language Models to Reason 16 Dec 2022
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn
Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, Deepak Ramachandran
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, Zhiting Hu
Reasoning Like Program Executors 27 Jan 2022
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Qiang Fu, Yan Gao, Jian-Guang Lou, Weizhu Chen
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, Vedant Misra
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, Behnam Neyshabur
Scaling Instruction-Finetuned Language Models 20 Oct 2022
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, Jason Wei
Galactica: A Large Language Model for Science 16 Nov 2022
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, Robert Stojnic
Ping Yu, Tianlu Wang, Olga Golovneva, Badr Alkhamissy, Gargi Ghosh, Mona Diab, Asli Celikyilmaz
STaR: Bootstrapping Reasoning With Reasoning 28 Mar 2022
Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman
Patrick Haluptzok, Matthew Bowers, Adam Tauman Kalai
Large Language Models Can Self-Improve 20 Oct 2022
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han
Arkil Patel, Satwik Bhattamishra, Navin Goyal
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, Sameer Singh
Jie Huang, Hanyin Shao, Kevin Chen-Chuan Chang
Large Language Models Still Can't Plan (A Benchmark for LLMs on Planning and Reasoning about Change) 21 Jun 2022
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, Subbarao Kambhampati
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, Behnam Neyshabur
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Shafiq Joty, Alexander R. Fabbri, Wojciech Kryscinski, Xi Victoria Lin, Caiming Xiong, Dragomir Radev
Abulhair Saparov, He He
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei
Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, Edward Grefenstette
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, Huan Sun
If you find this repo useful, please kindly cite our survey:
@article{huang2022towards,
title={Towards Reasoning in Large Language Models: A Survey},
author={Huang, Jie and Chang, Kevin Chen-Chuan},
journal={arXiv preprint arXiv:2212.10403},
year={2022}
}