A curated list of open-source instruction tuning datasets, models, papers, repositories.

Following Longpre et al., we list all existing instruction tuning datasets modified from traditional NLP tasks.

Most existing datasets are in English. However, most of the world’s population is under-served in terms of availability of data for their languages. How to ensure that everyone across the world is able to benefit from generative AI ? We have developed a straightforward and open-source translation tool based on Helsinki-NLP, capable of translating English datasets into 100+ languages at no cost. Although these translated datasets may contain some noise, they serve as a viable alternative to costly, high-quality data. See below.

python  translator.py  model_name  source_data_path

python  translator.py  Helsinki-NLP/opus-mt-en-zh  alpaca_data.json

Our tool is designed to work with alpaca data and the Helsinki-NLP/opus-mt-en-zh model. Different datasets or Helsinki-NLP models yield varying results. Due to the limitations of the model, Constrained by the model's capabilities, the translation quality may not always be optimal. For example，we observed instances of repeated words in the translations from English to Chinese，which lead us to develop "process.py" to eliminate translated prompts containing strings of any length that appear three consecutive times. We provide the final version in "translated_alpaca_data.json".

python  process.py  unprocessed_data_path

python  process.py  translated_data.json

# the Helsinki-NLP model may have a maximum input sentence length limit. We have discarded the prompts which exceed the limit before translate them.

We have extensively reviewed papers in this field and have listed the most valuable ones below:

Finetuned language models are zero-shot learners 2021.9

Multitask Prompted Training Enables Zero-Shot Task Generalization 2021.10

Training language models to follow instructions with human feedback 2022.3

Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks 2022.4

Unsupervised Cross-Task Generalization via Retrieval Augmentation 2022.4

Instruction Induction: From Few Examples to Natural Language Task Descriptions 2022.5

Scaling Instruction-Finetuned Language Models 2022.10

Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners 2022.10

Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor 2022.12

Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations 2022.12

Self-Instruct: Aligning Language Model with Self Generated Instructions 2022.12

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning 2022.12

The Flan Collection: Designing Data and Methods for Effective Instruction Tuning 2023.1

In-Context Instruction Learning 2023.2

Additionally, we have provided a list of related repositories for further reference.

awesome-instruction-learning

awesome-instruction-dataset

ICL_PaperList

prompt-in-context-learning

LM-reasoning

LLM-Reasoning-Papers

Chain-of-ThoughtsPapers

OpenICL