Scripts for fine-tuning Llama 2 using the Hugging Face TRL library

pytroch-cuda version can be collected from anaconda pytroch-cuda

conda install pytorch torchvision torchaudio pytorch-cuda=<cuda-version>12.1 or other latest version</cuda-version> -c pytorch -c nvidia

pip install -U -r requirements.txt
pip install -U datasets

If you see the error from the training script. LlamaConverter requires the protobuf library but it was not found in your environment. Checkout the instructions on the installation page of its repo: https://github.com/protocolbuffers/protobuf/tree/master/python#installation and follow the ones that match your environment. Please note that you may need to restart your runtime after installation.

pip install protobuf

7b and 13b models are able to be SFT and DPO under a single 4090. The 7b model should be able to fit in one 4080 for DPO depending on your LoRa config.

python sft_trainer.py 

Update the adapter path in merge_peft_adapters.py and run the script to merge peft adapters back to pretrained model. Note that the script is hardcoded to use CPU to merge the model in order to avoid CUDA out of memory errors. However, if you have sufficient VRAM on your GPU, you can change it to use GPU instead.

python merge_peft_adapters.py

python dpo_trainer.py 

Update the script generate.py and run the script to check the fine-tuned model output.

python generate.py

For the 7b or 13b model, because it has the same architecture as the Llama 1 model, you would follow the readme in https://github.com/qwopqwop200/GPTQ-for-LLaMa or https://github.com/PanQiWei/AutoGPTQ. But for 34b or 70b models, you may have to use autoGPTQ.