LlamaFactory

Experiments on APPS dataset¶

To test the performance of SFT training of LlamaFactory, we run experiments on the APPS dataset. We conduct SFT training on the APPS training dataset, and test the model on APPS test dataset. Below we list configurations of our experiment.

Dataset source: https://huggingface.co/datasets/codeparrot/apps/tree/main

Model: deepseek-coder-6.7b-instruct

Requirements: We use Python 3.10.13.

1. Clone source code:

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
# We use version 0.8.3.dev0
git checkout bda302fbfbdb114dee7782d405732600d2d73279

2. Run pip install -r assets/LlamaFactory/requirements.txt
3. Download flash_attn and install: https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.0/flash_attn-2.5.0+cu122torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
4. Install LlamaFactory:

cd LLaMA-Factory
pip install -e ".[torch,metrics]"

Training procedure¶

I run the experiments on 8 A100-PCIE-40GB GPUs (r8a100-a02). The detailed training config is in assets/LlamaFactory/dscoderinst_full_sft_ds3.yaml, please copy it to examples/train_full in LlamaFactory repo.

The bash command for training:

llamafactory-cli train examples/train_full/dscoderinst_full_sft_ds3.yaml

The total training time for the two epochs are:

Training with packing: 3h33min21s.

Training without packing: 8h58min33s.

Training loss¶

Training with packing:

Last step loss: 0.3562

Training without packing:

Last step loss: 0.3207

Evaluation¶

Currently, a total of 3,765 problems from the test set are evaluated, with problems lacking solutions filtered out. For each problem, 10 pieces of code are generated. The sampling temperature is set to 0.6, and top_p is set to 0.95.

The inference stage is run on 8 GPUs (r8a100-a02) with the following command:

python -m llmkit_data.cli.sample --prompts $DATAFILE_PATH --out $SAMPLE_PATH --model $MODEL_PATH --gpu_per_model 1
python -m llmkit_data.cli.eval_apps --samples $SAMPLE_PATH --out $RESULT_PATH --apps $APPS_PATH

Inference is conducted using vllm, taking 16min35s for code generation, and approximately 20 minutes for evaluation.

Results:¶

The pass@1, pass@5, and pass@10 statistics across different difficulty levels are presented in the table below:

Training with packing:

Training without packing:

For comparison, there is an experiment for the original deepseek-coder-6.7b-instruct model.