LLM Foundry is the open-source training codebase behind Databricks’ foundation models, including the MPT family and DBRX. It provides the code to train, fine-tune, evaluate and deploy large language models efficiently, built on MosaicML’s Composer library.
The repository is organized around the workflows model teams actually run: data preparation scripts that convert raw text into streaming dataset format, training scripts for Hugging Face and MPT models from 125M to 70B parameters, inference utilities that convert checkpoints to Hugging Face or ONNX format, and evaluation harnesses for in-context-learning tasks. It is the codebase that produced MPT models โ GPT-style architectures with Flash Attention, ALiBi for context-length extrapolation and stability fixes โ and DBRX, Databricks’ 132B-parameter mixture-of-experts model. Throughput and MFU benchmarking tools are included, along with prebuilt Docker images and support for NVIDIA A100 and H100 clusters, with beta paths for AMD and Intel Gaudi hardware. LLM Foundry is Apache-2.0 licensed, has more than 4,400 stars on GitHub, and remained under active development in 2026 as Databricks’ model training stack evolved. A registry system lets teams plug in custom models, datasets, callbacks and algorithms without forking the codebase, which is how many organizations adapt it to their own training pipelines.
Free. LLM Foundry is open source under the Apache-2.0 license; you provide your own GPU infrastructure, or optionally run workloads through Databricks’ platform.
ML engineers and research teams pretraining or fine-tuning their own foundation models, especially those who want a battle-tested, Composer-based training stack rather than a from-scratch training loop.