TorchScale is a Microsoft research library that provides the transformer architecture building blocks behind a family of foundation models for language, vision, speech and multimodal tasks. Published on GitHub under the MIT license with roughly 3,140 stars, it was last actively updated in 2024. Where most frameworks give you a training loop, TorchScale focuses on the architectures themselves: reusable, well-tested implementations of the model designs from Microsoft’s General AI research.
The library implements components such as multiway transformers that route tokens through modality-specific experts, the Magneto architecture with its improved training stability, and building blocks used by models in the BEiT and BEiT-3 line. It covers pretrained language models, vision transformers, translation and speech processing architectures, and exposes them in a modular PyTorch style so researchers can assemble variants, reproduce paper results, or graft individual components into their own models.
Several of the designs in TorchScale underpin well-known models. The multiway transformer, in which different experts process different modalities, is the backbone of BEiT-3’s unified vision-language pre-training, while Magneto introduced architectural changes such as sub-layer normalisation ordering that improve training stability at scale. TorchScale grew out of Microsoft’s earlier UniLM work and is structured as a research platform: components are meant to be mixed, matched and ablated, and the repository includes configurations used to reproduce results from the associated papers.
TorchScale is completely free and open source under the MIT license. There is no commercial tier; any costs are your own training compute.
ML researchers and engineers who want trustworthy implementations of modern transformer architectures to reproduce results or use as components in their own foundation-model work.