Transformer Engine

Transformer Engine is NVIDIA’s library for accelerating transformer models on modern NVIDIA GPUs, using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell hardware to train and serve large models faster with less memory. It is the low-level engine behind much of the ecosystem’s low-precision training, integrated into frameworks such as Megatron-LM, NeMo and DeepSpeed. The repository has around 3,600 stars on GitHub and is Apache-2.0 licensed.

Overview

Transformer Engine provides drop-in accelerated modules — linear layers, layer norms, attention and fused kernels — for PyTorch and JAX, together with the recipes needed to keep low-precision training stable, such as per-tensor scaling and FP8 formats tuned per layer type. By doing matrix multiplications in FP8 or FP4 while keeping sensitive operations in higher precision, models can roughly double throughput and halve memory on supported GPUs without meaningful accuracy loss. The library handles the details of scaling-factor management and casts automatically, and exposes higher-level APIs so framework users can enable it with minimal code changes.

Key Features

• FP8 and FP4 accelerated transformer layers for PyTorch and JAX
• Fused kernels for linear, normalisation and attention operations
• Automatic scaling recipes that keep low-precision training stable
• Major speed and memory gains on Hopper, Ada and Blackwell GPUs
• Integrated by Megatron-LM, NeMo, DeepSpeed and other frameworks
• C++ core with Python APIs for both training and inference
• Apache-2.0 licensed

💳 Pricing

Transformer Engine is completely free and open source under the Apache-2.0 license. The real requirement is compatible NVIDIA GPU hardware to run it on.

Who Is It For

ML engineers and framework developers training or serving large transformers who want the throughput of low-precision compute without hand-writing kernels or debugging numerical instability themselves.