Transformer Deploy

Transformer Deploy is an open-source inference server for Hugging Face transformer models, developed by ELS-RD (the R&D team at Lefebvre Sarrut). It converts encoder models to ONNX and TensorRT, then serves them behind an efficient HTTP API on CPU or GPU with the throughput and latency profile of an enterprise deployment rather than a research script. The project has around 1,700 stars and was last updated in 2024.

Overview

Serving BERT-style models in production is mostly an optimisation problem: graph conversion, precision tuning and batching matter more than the model itself. Transformer Deploy packages that expertise into a repeatable workflow — point it at a Hugging Face model, and it produces an optimised TensorRT or ONNX Runtime server with benchmarking tools to verify the speed-up before you ship it.

Key Features

• One-command conversion of Hugging Face encoders to ONNX and TensorRT
• High-throughput inference server for CPU and NVIDIA GPU deployments
• Quantisation and precision options to trade accuracy for speed deliberately
• Built-in benchmarking to measure latency and throughput gains
• Simple HTTP API that drops into existing microservice architectures
• Apache-2.0 licensed, from the team running it in legal-tech production

💳 Pricing

Free. Transformer Deploy is open source under the Apache-2.0 license; you run it on your own infrastructure at no cost.

Who Is It For

ML platform engineers productionising transformer models, teams moving from prototype notebooks to served APIs, and anyone who needs BERT-class inference to be fast and cheap at scale.

The project documents end-to-end examples for common encoder workloads such as text classification and token classification, and its maintainers report substantial latency reductions versus naive PyTorch serving in the benchmarks published with the code.