nanoT5 is a fast, simple repository for pre-training and fine-tuning T5-style encoder-decoder models in PyTorch, built in the spirit of nanoGPT but for the T5 family. It demonstrates that a full pre-training run is achievable on a modest budget: a single A100 GPU in under a day. The project is Apache-2.0 licensed and has earned over 1,000 stars on GitHub.
The repository reproduces the pre-training of T5-base v1.1 (248M parameters) from random initialization on the English subset of the C4 dataset, followed by fine-tuning on the Super-Natural Instructions benchmark, reaching about 40.7 ROUGE-L in roughly 16 hours on one GPU. The codebase favors clarity over framework magic, making each stage — data processing, the training loop, and fine-tuning — easy to read, hack, and extend. A published paper and detailed write-ups accompany the code, documenting the setup, the compute budget, and the conclusions, so others can replicate the recipe or adapt it to their own encoder-decoder experiments without reverse-engineering a large framework.
The budget framing is the point of the project: by publishing exact hardware, time, and cost figures alongside the code, nanoT5 turns pre-training from a datacenter ritual into an experiment a graduate student can actually run. That transparency makes it useful for studying training dynamics — learning-rate schedules, masking strategies, and data mixtures can be varied and compared within days rather than months. It also serves as a template for encoder-decoder work in domains like summarization and translation, where T5-style models remain competitive.
nanoT5 is free and open source under the Apache-2.0 license; the only cost is your own GPU time.
It is aimed at students, researchers, and engineers who want to understand or reproduce LLM pre-training hands-on, without a datacenter budget.