Firefly is a widely used open-source toolkit for training and fine-tuning large language models, with ready recipes for more than thirty open model families — Qwen, Llama, Gemma, ChatGLM, Baichuan, Mistral, MiniCPM and many more. It has earned about 6,600 stars on GitHub and was actively developed through 2024, becoming a go-to training codebase in the Chinese open-source community.
Fine-tuning a modern LLM involves a pile of near-identical training scripts that differ mainly in tokenizers, templates and hyperparameters. Firefly standardizes that work: pick a base model, point it at your instruction data, choose full-parameter training or a parameter-efficient method such as LoRA or QLoRA, and train. It covers the full pipeline — pretraining, supervised fine-tuning and preference optimization such as DPO — and ships curated open instruction datasets plus the conversation templates each model family expects. Because the same harness supports so many architectures, experiments learned on one model transfer directly to the next release. Firefly is built on PyTorch and the Hugging Face ecosystem, runs on a single multi-GPU machine, and its documentation and community discussions are primarily in Chinese, with code and configs that English-speaking users also navigate comfortably.
Free. Firefly is an open-source project you run on your own hardware; the only costs are your GPUs and the base models you choose to train.
ML engineers and researchers — especially in the Chinese open-source ecosystem — who fine-tune open LLMs and want a single, proven training harness instead of rewriting scripts for every new model release.