PatrickStar

PatrickStar is an open-source parallel training system from Tencent that makes it possible to train larger pretrained models, faster and on less hardware, through a chunk-based memory management design. It tackles the out-of-memory wall that stops many teams from training large language models, with the stated goal of democratizing large-model training beyond a handful of well-funded labs. The project is hosted on GitHub, where it has earned roughly 770 stars, and it is still maintained, with updates in 2025.

Overview

Training a large model fails most often not on compute but on memory: parameters, gradients and optimizer states multiply fast. PatrickStar manages model data in chunks and dynamically orchestrates that data across a heterogeneous memory space of GPU and CPU memory, so the GPU only holds what it actively needs. Built on PyTorch, this approach lets a single machine — or a modest cluster — train models that would otherwise require far more GPUs, and it improves hardware utilization compared with naive offloading. The system was designed for BERT- and GPT-style pretraining workloads and is described in an accompanying research paper from the Tencent team.

Key Features

• Chunk-based memory management for model parameters, gradients and optimizer states
• Heterogeneous GPU/CPU memory orchestration to break the GPU memory wall
• Train larger pretrained models on fewer GPUs, lowering the entry barrier
• Built on PyTorch for BERT, GPT and similar NLP pretraining workloads
• Dynamic data placement that improves memory utilization over static offloading
• Backed by a published paper and maintained by Tencent engineers

💳 Pricing

PatrickStar is free and open source under the BSD-3-Clause license. There is no paid edition or hosted training service attached to the project.

Who Is It For

PatrickStar is for ML engineers and research groups who pretrain or fine-tune large models and are constrained by GPU memory — teams that need big-model training to fit the hardware they actually have.