loralib is Microsoft’s original implementation of LoRA — Low-Rank Adaptation of Large Language Models — the fine-tuning technique that changed how the industry adapts big models. The repository has more than 13,800 stars on GitHub and was most recently updated in late 2024.
Instead of updating every weight in a pretrained model, LoRA freezes the original weights and learns pairs of small rank-decomposition matrices for each adapted layer. The result matches full fine-tuning quality on benchmarks such as GLUE with RoBERTa, DeBERTa and GPT models, while training and storing only a fraction of the parameters — and because the adapters merge back into the model, they add no inference latency. This PyTorch package is the reference release from the paper’s authors; the technique has since also been adopted by Hugging Face’s PEFT library.
In practice, adopting loralib means adding trainable low-rank matrices alongside the attention and feed-forward projections of a transformer, training only those matrices on task data, and then either merging them into the base weights for deployment or keeping them as tiny, swappable adapter files — often just a few megabytes, even for very large models. The paper demonstrated adaptation of models up to GPT-3 scale on a single GPU. LoRA has since become the default parameter-efficient fine-tuning method across the ecosystem, which makes this repository — the authors’ own reference code, with worked examples — a common starting point for understanding how it actually behaves.
loralib is free and open source under the MIT license, installable from PyPI. Your only costs are the compute for training and serving your models.
ML engineers fine-tuning large models on limited hardware, researchers studying parameter-efficient adaptation, and teams serving many task-specific adapters from one base model.