Megatron-LM

Megatron-LM is NVIDIA’s pioneering open-source codebase for training transformer language models at very large scale, and the origin of the model-parallelism techniques the whole industry now uses. The repository pairs the Megatron-LM reference training examples with Megatron Core, a composable library of GPU-optimized building blocks for custom training frameworks. It has earned about 18,200 stars on GitHub and remains in heavy development in 2026.

Overview

Megatron Core provides transformer building blocks, advanced parallelism strategies (tensor, pipeline, data, expert and context parallelism), mixed-precision support from FP16 and BF16 down to FP8 and FP4, and optimized kernels and communication overlap. Megatron-LM wraps it with pre-configured scripts for training GPT-style, hybrid Mamba and vision-language models, plus reinforcement learning and post-training tools such as quantization and distillation. The codebase has benchmarked a 462-billion-parameter model across 6,144 H100 GPUs at up to 47% model FLOP utilization, and Megatron Bridge converts checkpoints to and from Hugging Face formats with production recipes for models like DeepSeek-V3 and Qwen3.

Key Features

• Composable Megatron Core library with GPU-optimized transformer building blocks
• Five-dimensional parallelism: tensor, pipeline, data, expert and context parallelism
• Mixed precision down to FP8 and FP4, with fine-grained communication overlap
• Reference training for GPT, Mamba hybrids and vision-language models at up to 462B parameters
• Megatron Bridge for bidirectional Hugging Face checkpoint conversion with ready recipes
• Post-training tooling for quantization, distillation, pruning and RL fine-tuning

💳 Pricing

Megatron-LM and Megatron Core are free and open source from NVIDIA. You provide the GPU clusters; there are no licensing fees for the library itself.

Who Is It For

Megatron-LM is for framework developers and ML research teams training frontier-scale language models who need proven, high-performance parallelism and full control over the training stack.