Mesh Transformer JAX is an open-source implementation of model-parallel transformers in JAX and Haiku, best known as the codebase used to train GPT-J-6B. It has collected about 6,400 stars on GitHub, is licensed under Apache-2.0, and its code was last updated in 2023.
Training language models with billions of parameters requires splitting the model itself across many accelerators, not just the data. Mesh Transformer JAX tackles this with mesh parallelism: transformer layers are sharded across a mesh of TPU or GPU cores using JAX’s pmap and xmap abstractions, so very large GPT-style models fit in aggregate memory and train efficiently. The project was developed within the EleutherAI ecosystem and proved the approach by producing GPT-J-6B, one of the first strong open GPT-3-class models, making it a historically important reference implementation.
The project’s historical impact extends well beyond its own checkpoints. GPT-J-6B, trained with this codebase on TPU infrastructure, was for a time the strongest openly available GPT-3-class model and powered a wave of open research and applications. The codebase also served as a teaching tool for a generation of engineers learning distributed deep learning, because its mesh-parallelism logic is compact enough to read end to end while still demonstrating the real techniques used at scale.
Mesh Transformer JAX is free and open source under the Apache-2.0 license. There is no paid edition; your only costs are the TPU or GPU compute you run it on.
Mesh Transformer JAX is for ML researchers and engineers studying distributed training who want a compact, readable example of how model parallelism works in JAX, or a foundation for large language model experiments.